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12th  ANNUAL  SYSTEMS  ENGINEERING  CONFERENCE 

“Achieving  Acquisition  Excellence  Via  Effective  Systems  Engineering  ” 

San  Diego,  CA 
26  -  29  October  2009 


Agenda 


Tuesday,  October  27,  2009 

PLENARY  SESSION  1  -  INTRODUCTION  &  OPENING  REMARKS 

•  Mr.  Bob  Rassa,  Director,  Systems  Supportability,  Raytheon;  Chair,  Systems  Engineering  Division,  NDIA 

KEYNOTE  ADDRESS 

•  Honorable  Zachary  J.  Lemnios,  Director,  Defense  Research  and  Engineering,  Office  of  the  Secretary  of  Defense  (Acquisition,  Technology  and 
Logistics) 

PLENARY  SESSION  2  -  SERVICE  ACQUISITION  EXECUTIVES 

VIEW  FROM  THE  TOP:  HOW  CAN  SE  SUPPORT  PROGRAM  EXECUTION? 

Moderator:  Mr.  Terry  Jaggers,  Principal  Deputy,  Systems  Engineering,  Office  of  the  Director,  Defense  Research  and  Engineering 

•  Mr.  David  G.  Ahem,  Director,  Portfolio  Systems  Acquisition,  Office  of  the  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics) 

•  Mr.  Thomas  E.  Mullins,  Deputy  Assistant  Secretary  for  Plans,  Programs  and  Resources  (SAAL-ZR),  Office  of  the  Assistant  Secretary  of  the 
Army  (Acquisition,  Logistics  and  Technology) 

•  Mr.  Christopher  A.  Miller,  PEO  for  Command,  Control,  Communications,  Computers  and  Intelligence  (C4I),  ASNRDA 

•  Mr.  Randall  G.  Walden,  Director,  Information  Dominance  Programs,  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition),  AFRCO 

LUNCH  WITH  SPEAKER  IN  THE  REGATTA  PAVILION 

•  Mr.  Stephen  Welby,  Director,  Systems  Engineering,  Office  of  the  Director,  Defense  Research  and  Engineering 

PLENARY  SESSION  3  -  TEST  &  EVALUATION  EXECUTIVES 

VIEW  FROM  THE  TOP:  HOW  SE  CAN  SUPPORT  DEVELOPMENTAL  TEST  AND  EVALUATION? 

Moderator:  Mr.  Jim  O’Bryon,  The  O’Bryon  Group;  Chair,  Test  and  Evaluation  Division 

•  Dr.  James  N.  Streilein,  Technical  Advisor,  HQ  Army  Test  &  Evaluation  Command 

•  Ms.  Amy  Markowich,  Deputy  DoN  T&E  Executive 

•  Colonel  Dexter  M.  Sapinoso,  USAF,  Chief  of  Air  Force  Test  and  Evaluation  Policy  and  Programs 

•  Mr.  Christopher  DiPetto,  OU SD( AT &L)/DDR&E/DT &E 

PLENARY  SESSION  4  -  SE  AND  ACQUISITIONS  REFORM:  THE  WAY  AHEAD 

Moderator:  Mrs.  Kristen  Baldwin,  Office  of  the  Director,  Defense  Research  and  Engineering 

•  Mr.  Ross  Guckert,  Office  of  the  Secretary  of  the  Army  for  Acquisition,  Logistics  and  Technology  (ASA(ALT)) 

•  Mr.  Carl  Siel,  Office  of  the  Assistant  Secretary  of  the  Navy  for  Research,  Development  and  Acquisition  (ASN(RDA)CHSENG) 

•  Colonel  Shawn  Shanley,  USAF,  Chief  Systems  Engineer,  Office  of  the  Assistant  Secretary  of  the  Air  Force  for  Acquisition,  Science, 
Technology,  and  Engineering  (SAF/AQR) 

•  Mr.  Nicholas  Torelli,  Office  of  the  Director,  Defense  Research  and  Engineering 


WEDNESDAY.  OCTOBER  28.  CONCURRENT  SESSIONS 
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TRACK  1 

Systems  Engineering  Effectiveness  -  Bayview  III 

•  885 1  -  Rapid  Development  and  Integration  of  Remote  Weapon  Systems  to  Meet  Operational  Requirements ,  Mr.  Joseph  Burkart,  NSWC  Crane, 
Small  Arms  Air  Platform  Integration 

•  8893  -  Rapid  Development,  Mr.  Michael  Gaydar,  NAVAIR 

•  8847  -  Tailoring  the  SE  Process  to  Effectively  Complement  the  SW  Agile  Development  Process ,  Mr.  William  Lyders,  AS  SETT  Inc. 

•  8902  -  Systems  Engineering  Leading  Indicators:  Insight  into  Effective  Systems  Engineering ,  Mr.  Gary  Roedler,  Lockheed  Martin  Corporation 

•  9414  -  Correcting  Deficiencies  in  the  Systems  Engineering  of  Tactical  Weapons ,  Mr.  Marvin  Ebbert,  Raytheon  Missile  Systems 

•  8948  -  Value  Engineering  Applications  in  Service  Contracts ,  Dr.  Jay  Mandelbaum,  Value  Engineering  Applications  in  Service  Contracts 

•  88 16  -  Mind  the  GAPs-a  Systems  Engineering  Implementation  of  DoDI  5000. 02,  Dr.  Thomas  Christian,  U.  S.  Air  Force 

•  8990  -  Systems  Engineering  for  Rapid  Capability  Development ,  Mr.  Thomas  McDermott,  Georgia  Tech  Research  Institute 

•  8974  -  Transforming  Systems  and  Software  Engineering  Across  an  Enterprise,  Mr.  Jeffery  Wilcox,  Lockheed  Martin  Corporation 

•  8863  -  Using  Requirements  Compliance  to  Identify  Gaps  Between  the  Technical  Solution  and  Requirements,  Mr.  Frank  Salvatore,  High 
Performance  Technologies,  Inc. 

•  8823  -  Win  and  Influence  Design  Engineers — Change  Their  Affordability  DNA,  Mr.  Tim  Morrill,  Raytheon  Company 

TRACK  2 

Early  System  Engineering  -  Bayview  II 

•  895 1  -  USAF  View  ofNRC  “Pre-A  Systems  Engineering ”  Study  Committee  Recommendations  As  Addressed  By  Levin-McCain  (P.L.  111-23; 

“Weapon  Systems  Acquisition  Reform  Act  of 2009”),  Mr.  Jeff  Loren,  SAF/AQR  (Alion  Science  &  Technology) 

•  8846  -  Air  Force  Materiel  Command  Early  Systems  Engineering,  Dr.  Brian  Kowal,  USAF 

•  9016-^4  Framework  for  Enhancing  Forward-looking  Capability  Delivery  Metrics,  Mr.  Leonard  Sadauskas,  DoD  CIO  CT&S 

•  9082  -  Including  Environment,  Safety,  and  Occupational  Health  (ESOH)  Requirements  in  Joint  Capabilities  Integration  and  Development 
System  (JCIDS)  Documents,  Mr.  Sherman  Forbes,  U.S.  Air  Force 

•  8835  -  T&E  Collaboration  and  Contributions  during  Early  Program  Acquisition,  Mr.  Stephen  Scukanec,  Northrop  Grumman  Corporation 
Aerospace  Systems 

•  8795  -  Mission-based  Test  and  Evaluation  Strategy:  Creating  Linkages  between  Technology  Development  and  Mission  Capability,  Mr.  John 
Beilfuss,  U.S.  Army  Research  Laboratory 

•  Panel  Topic:  8924,  8925 , 8933  -  Early  Systems  Engineering  in  DoDI  5000.02,  Dr.  Judith  Dahmann,  Ms.  Lisa  Reuss,  Ms.  Sharon  Vannucci, 
Systems  Engineering  Directorate,  ODDR&E 

•  8949  -  Updated  DoD  5000  and  CJCS  31 70  Policies:  A  Requirements  to  Acquisition  Gap  Analysis,  Mr.  John  Lohse,  Raytheon  Company 

•  88 1 3  -  Emerging  Roles  for  Systems  Engineering  in  Defense  Decision  Making;  Better  Aligning  Requirements  and  Acquisition  with  the  Budget 
and  Security  Environments,  Mr.  Vincent  Roske,  Institute  for  Defense  Analyses 

•  9026  -  Early  SE  Determination  of  Best-Fit  System  Life  Cycle  Processes,  Dr.  Barry  Boehm,  USC 

TRACK  3  -  Technology  Maturity  -  Bayview  I 

•  8991  -  Systems  Engineering  for  the  Science  &  Technology  Community,  Mr.  Russell  Menko,  U.  S.  Army  RDECOM/TARDEC 

•  9017  -  Linking  Systems  Engineering  Artifacts  with  Complex  Systems  Maturity  Assessments,  Dr.  Brian  Sauser,  Stevens  Institute  of  Technology 

•  8770  -  Incorporating  Maturity  Assessment  into  House  of  Quality  for  Improved  Decision  Support  Analysis  and  Risk  Management,  Mr.  Pavel 
Fomin,  U.S.  Air  Force 

•  8798  -  The  New  Technology  Readiness  Assessment  Process,  Dr.  Jay  Mandelbaum,  Institute  for  Defense  Analyses 

•  8870  -  S&T  Portfolio  Maturity  &  Performance  Analysis:  The  Concept  of  Critical  Research  Elements,  Mr.  Has  Patel,  Infologic,  Inc. 

•  8879  -  TRL  Vectors  in  IPPD-based  Portfolio  Management,  Mr.  Michael  Bartmess,  General  Dynamics/ AIS 

•  8963  -  Air  Force  Concept  Maturity  Assessment,  Mr.  George  Freeman,  U.S.  Air  Force,  Center  for  Systems  Engineering 

•  8900  -  DOD ’s  Weapon  System  Portfolio:  Are  Results  Getting  Any  Better?,  Mr.  Michael  Sullivan,  U.S.  Government 

•  8894  -  Air  Force  Initiative  -  High  Confidence  Technology  Transition  Planning  Through  the  Use  of  Stage-Gates  -  Update,  Mr.  Randy  Bullard, 
U.S.  Air  Force  Materiel  Command 

•  8891  -  A  comprehensive  overview  of  techniques  for  measuring  system  readiness,  Mr.  James  Bilbro,  JB  Consulting  International 

TRACK  4  -  Test  and  Evaluation  of  SOS  -  Mission  I 

•  8825  -  Test  and  Evaluation  in  a  System  of  Systems  Environment,  Mr.  Edwin  McDermott,  653  ELSW,  Electronic  Systems  Center 

•  8849  -  Joint  Integration  and  Interoperability  Lab  (JSIIL),  Mr.  Steven  Whitehead,  SL,  J8  Technical  Director,  USJFCOM 

•  8935  -  Systems  of  Systems  Systems  Engineering  and  Test  and  Evaluation,  Dr.  Judith  Dahmann,  Systems  Engineering  Directorate, 
ODDR&E/MITRE 

TRACK  4  -  Practical  Systems  Engineering  -  Mission  I 

•  9014  -  SAVI:  Aerospace  Platform  Development  and  Certification  Using  Modeling  and  Simulation  to  “Integrate,  then  Build”,  Mr.  Gregory 
Pollari,  Rockwell  Collins 

•  8855  -  Certify  and  Fly  Right:  Preparing  for  DO-297  Certification,  Mr.  Ketih  Custer,  Esterline  Control  Systems- A  VIST  A 

•  8973  -  C-17  Transition  to  Criteria-based  Airworthiness  Certification,  Mr.  Christian  Stillings,  USAF  516  AESG 

TRACK  4  -  Test  and  Evaluation  -  Mission  I 

•  8848  -  Integrated  Testing:  We  Can  Do  It,  Dr.  Beth  Wilson,  Raytheon  Company 

•  8882  -  Test  &  Evaluation  Strategy  for  the  Technology  Development  Phase,  Ms.  Darlene  Mosser-Kemer,  OUSD(AT&L)/DDR&E/DT&E 

•  8883  -  Test  &  Evaluation  Products  for  the  Systems  Engineering  Reviews,  Mr.  Woody  Eischens,  OUSD(AT&L)/DDR&E/DT&E 

•  88 14  -  Joint  Mission  Environment  Test  Capability  (JMETC),  Lowering  technical  Risk  by  Improving  Distributed  Test  Capabilities,  Mr.  Chip 
Ferguson,  JMETC 

•  8901  -  Review  Results  of  the  NDIA/OSD  Software  Test  Summit/Workshop,  Mr.  Thomas  Wissink,  Lockheed  Martin  IS&GS 

TRACK  5  -  Human  Systems  Integration  -  Mission  II 

•  8998  -  Human  Systems  Integration  -  Ensuring  the  Human  is  Considered  “Left  of  A  ”,  Col  Larry  Kimm,  USAF,  U.S  Air  Force 

•  8885  -  Human  Systems  Integration  (HSI)  -  Integrating  Human  Concerns  into  Life  Cycle  Systems  Engineering,  Ms.  Cynthia  Shewed,  Booz  Allen 
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Hamilton 

•  9012  -  Human  Systems  Integration:  Defining  and  Validating  a  Framework  for  Enhanced  Systems  Development,  Dr.  Matthew  Risser,  Pacific 
Science  &  Engineering  Group 

•  8975  -  What  is  Human  Systems  Integration  (HSI)  and  why  should  we  do  it?,  Mr.  Stuart  Booth,  Systems  Engineering  Directorate,  ODDR&E 

•  9042  -  Bounding  the  Human  Within  the  System,  Mr.  Michael  Mueller,  U.S.  Air  Force  Center  for  Systems  Engineering 

•  8829  -  The  Army  Health  Hazard  Assessment  Program ’s  Medical  Cost  Avoidance  Model,  Dr.  Timothy  A.  Kluchinsky,  Department  of  Army 

TRACK  5  -  System  Safety  -  ESOH  -  Mission  II 

•  9095  -  Using  Proposed  MIL-STD-882  Change  1  For  Hazardous  Materials  Management,  Ms.  Karen  Gill,  Booz  Allen  Hamilton 

•  8890  -  Building  Safer  UGVs  with  Run-time  Safety  Invariants,  Mr.  Michael  Wagner,  Carnegie  Mellon  University,  NREC 

•  Sherman  Forbes,  U.S.  Air  Force 

•  882D  -  Overview  of  Draft  MIL-STD-882D  With  Change  1,  Mr.  Bob  Smith,  Booz  Allen  Hamilton 

•  9070  -  Improving  Safety  Technology  Insertion  in  DoD  Acquisition  Programs ,  Dr.  Elizabeth  Rodriguez- Johnson,  Systems  Engineering 
Directorate,  ODDR&E 

•  9094  -  DoD  Green  Procurement  Program  Update  and  Path  Forward ,  Mr.  David  Asiello,  Office  of  the  Secretary  of  Defense 

•  909 1  -  Environment,  Safety,  and  Occupational  Health  (ESOH)  Risk  and  Technology  Requirements  Reporting  at  Acquisition  Program  Reviews , 
Ms.  Lucy  Rodriguez,  Booz  Allen  Hamilton 

TRACK  6  -  System  of  Systems  -  Mission  III 

•  8898  -  Designing  Collaborative  Systems  of  Systems  in  support  of  Multi-sided  Markets ,  Mr.  Philip  Boxer,  SEI 

•  8892  -  SysML  Strategies  to  Characterize  and  Analyze  Systems  of  Systems ,  Dr.  Jo  Ann  Lane,  University  of  Southern  California 

•  9041  -  On  Modeling  and  Simulation  Methods  for  Capturing  Emergent  Behaviors  for  Systems  of  Systems ,  Dr.  Jack  Zentner,  Georgia  Tech 
Research  Institute 

•  9060  -  M&S  Support  for  SoS  SE,  Dr.  Joann  Lane,  USC 

•  8776  -  The  Modular  SOS  Paradigm:  an  Availability  Paradox ?,  Mr.  Peter  Gentile,  Northrop  Grumman  Corporation 

•  8866  -  Extending  FMECA  to  Systems  of  Systems,  Mr.  Leopoldo  Mayoral,  Johns  Hopkins  University/ APL 

•  NDIA  SoS  Committee  Report ,  Dr.  Judith  Dahman,  Systems  Engineering  Directorate,  ODDR&E/MITRE 

•  8960  -  A  Distillation  of  Lessons  Learned  from  Complex  System  of  Systems  Acquisitions,  Dr.  Richard  Turner,  Stevens  Institute 

•  8784  -  Establishing  a  Departmental-Level  Systems-of-Sy stems  Engineering  Management  Construct  for  the  Department  of  the  Navy,  Progress 
Report ,  Mr.  John  Kevin  Smith,  Asst  Sec  of  the  Navy  for  RD&A,  Chief  Engineer 

•  8942  -  DoD  Systems  of  Systems  Update,  Dr.  Judith  Dahmann,  Systems  Engineering  Directorate,  ODDR&E/MITRE 

•  8961  -  Engineering  Systems  of  Systems:  An  Integration  Perspective,  Dr.  Emmett  Maddry,  NSWCDD 
TRACK  7  -  Program  Management  -  Palm  I 

•  8979  -  Boots  on  the  Ground:  Tactical  Planning  at  Program  Start  Up,  Mr.  Gerry  Becker,  Harris  Corporation 

•  8999  -  Program  Signature  Measurement,  Mr.  James  Thompson,  Systems  Engineering  Directorate,  ODDR&E 

•  9103  -  The  Economics  of  CMMI,  Mr.  Geoff  Draper,  Harris  Corporation 

•  8995  -  Integrated  Systems  Engineering  and  Developmental  Test  and  Evaluation,  Mr.  Chris  DiPetto,  DUSD(A&T)/SSE 

•  902 1  -  Critical  Success  Factors  for  Milestone  Review  Risk  Identification,  Dr.  Barry  Boehm,  USC 

•  9030  -  Lessons  Learned  in  Motivating  Software  Engineering  Process  Group  to  Focus  on  Achieving  Business  Goals  and  Not  on  Just  Achieving  a 
Maturity  Level,  Mr.  Girish  Seshagiri,  Advanced  Information  Services  Inc. 

•  9003  -  CMMI®  for  Executives,  Mr.  Geoff  Draper,  Harris  Corporation 

•  9034  -  Sustainment  and  Continued  Institutionalization  of  Best  Practices  and  CMMI®  at  SPAWAR,  Mr.  Michael  Kutch,  SPAWAR  Systems 
Center  Atlantic 

•  8791  -  Cost  and  Risk  Impacts  of  the  New  DOD  5000  Defense  Acquisition  Framework,  Dr.  Peter  Hantos,  The  Aerospace  Corporation 

•  8895  -  A  Comprehensive  Review  of  Maturity  Assessment  Approaches  for  Improved  Defense  Acquisition,  Ms.  Nazanin  Azizian,  The  George 
Washington  University 

TRACK  8  -  Net-Centric  Operations/Interoperability  -  Palm  II 

•  8874  -  The  Boeing  System  of  Systems  Engineering  (SoSE)  Process  and  Its  Use  in  Developing  Legacy-Based  Net-Centric  Systems  of  Systems,  Mr. 
John  Palmer,  The  Boeing  Company 

•  9010  -  Network  Enabled  Weapons,  A  System  Engineering  Approach  to  Achieve  Inter operabilty,  Mr.  Andrew  Lieux,  Naval  Air  Warfare  Center 
Weapons  Division 

•  8854  -  Human  Interoperability  Enterprise  and  Net  Centric  Operations,  Mr.  Jack  Zavin,  ASD  (Nil) 

•  8780  -  Net-Centric  Best  Practices ,  Mr.  Hiekeun  Ko,  JPEO-CBD  -  Software  Support  Activity 

•  8788  -  Data  sharing  in  a  Stability  Operations  Community  of  Interest:  Utilizing  a  pilot  program  to  prove  concepts  and  develop  trust.,  Mr.  Gerald 
Christman,  Femme  Comp  Inc. 

•  8853  -  C4I  Architecture  for  Joint  ASW,  Mr.  Gregory  Miller,  Naval  Postgraduate  School 

•  8929  -  Extending  Net-Centric  Quality  of  Service  to  Systems  of  Systems ,  Maj  Vinod  Naga,  USAF,  Air  Force  Institute  of  Technology 

•  9081  -  Testing  in  Service- oriented  Environments ,  Mr.  Soumya  Simanta,  SEI 

•  8913  -  Linking  Interoperability  and  Measures  of  Effectiveness:  A  Method  for  Evaluating  Architectures,  Dr.  David  Jacques,  Air  Force  Institute 
of  Technology 

TRACK  8  -  Speciality  Engineering  -  Palm  II 

•  8944  -  DoD ’s  Refocus  on  Specialty  Engineering  (Reliability,  Availability  and  Maintainability;  Producibility  and  Quality,  Supportability,  Safety 
and  Human  Systems  Integration),  Mr.  Chester  Bracuto,  Systems  Engineering  Directorate,  ODDR&E 

•  9043  -  Implementing  the  Materiel  Availability  KPP  in  DoD  acquisition  programs — balancing  life-cycle  costs  with  warfighter  needs,  Mr.  Grant 
Schmieder,  Systems  Engineering  Directorate,  ODDR&E 

•  8873  -  IUID  enables  streamlined  acquisition  and  system  engineering,  Mr.  Robert  Leibrandt,  DoD  UID  Policy  Office 

•  8958  -  Security  Systems  Engineering,  Mrs.  Kristen  Baldwin,  Systems  Engineering  Directorate,  ODDR&E 
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THURSDAY.  OCTOBER  29.  CONCURRENT  SESSIONS 

TRACK  1  -  Systems  Engineering  Effectiveness  -  Bayview  III 

•  8887  -  Achieving  a  Systems  Engineering  Culture  in  a  Science  and  Technology  Laboratory  Environment,  Mr.  Robert  Rapson,  Materials  and 
Manufacturing  Directorate,  AFRL 

•  8920  -  A  Methodology  for  Assessing  Systems  Engineering  Practices,  Ms.  Lauren  Levy,  Johns  Hopkins  University/ APL 

•  9097  -  Acquisition  ESOH  Risk  Management-How  to  Make  It  Work,  Mr.  Bob  Smith,  Booz  Allen  Hamilton 

TRACK  1  -  Architecture  -  Bayview  III 

•  883 1  -  Human-Centered  Design  in  Systems  Engineering:  Human  View  Methodology,  Dr.  Robert  Smillie,  SPAWAR 

•  8830  -  Systems  Engineering  Needs  of  the  DoDAF  -  Report  of  the  Architecture  Frameworks  Working  Group,  Mr.  Joe  Kuncel,  Northrop 

Grumman  Corporation 

•  8824  -  Delivering  DoDAF  Version  2. 0  to  Architects  and  Systems  Engineers  for  IT  Systems  and  Services ,  Mr.  Walt  Okon,  Department  of 
Defense,  CIO,  Enterprise  Architecture 

•  897 1  -  Advancing  Systems  Engineering  Practice  using  Model  Based  System  Development,  Mr.  Sanford  Friedenthal,  Lockheed  Martin 
Corporation 

•  9004  -  Evolving  Systems  Engineering  through  Model  Driven  Functional  Analysis,  Dr.  Mark  Blackburn,  Systems  and 

TRACK  2  -  Logistics  Systems  -  Bayview  II 

•  9063  -  An  Integrated  RAM  Approach  to  System  Design  and  Support,  Mr.  Robert  Finlayson,  Johns  Hopkins  University/ APL 

•  903 1  -  Supportability  Lessons  Learned  with  Line  Replaceable  Modules,  Ms.  Heity  Hsiung,  Raytheon  Company 

•  8908  -  Successful  First  AESA  Deployment  through  Application  of  System  Engineering,  Mr.  Scott  Nichols,  Raytheon  Company 

•  9039  -  Applying  Systems  Engineering  to  Fielded  Weapon  Systems  and  End-Items,  Mr.  Michael  Ucchino,  AF  Center  for  Systems  Engineering 

•  9008  -  Upgrade  Fluid  System  Filter  Element  Monitoring  To  Increase  Operational  Reliability  and  Support  Condition  Based  Maintenance 
Capability,  Mr.  Gary  Rosenberg,  Constellation  Technology  Corportation 

•  8834  -  Tailoring  Systems  Engineering  for  Technical  Support  of  Legacy  Products,  Mr.  Joseph  Skandera,  BAE  Systems 

•  9092  -  The  role  of  simulation  in  tracking  mobile  assets  using  RFID  technology,  Mr.  Swee  Leong,  National  Institute  of  Standards  and 
Technology 

TRACK  3  -  Modeling  &  Simulation  -  Bayview  I 

•  8939  -  Understanding  the  New  DoD  Instruction  5000.61:  ‘DoD  Modeling  &  Simulation  Verification,  Validation  and  Accreditation  (VV&A)  ”, 
Mr.  Michael  Truelove,  Systems  Engineering  Directorate,  ODDR&E 

•  8950  -  Live,  Virtual,  Constructive  Architecture  Roadmap:  The  Quest  for  Interoperability,  Standards,  and  Reuse,  Dr.  Gary  Allen,  Joint  Training 
Integration  &  Evaluation  Center 

•  9048  -  Revisions  to  the  Acquisition  Modeling  &  Simulation  Master  Plan,  Mr.  Stephen  Swenson,  Systems  Engineering  Directorate,  ODDR&E 

•  8759  -  A  Systems  Engineering  Framework  for  Integrating  M&S  Development  Best  Practices,  Dr.  Katherine  Morse,  Johns  Hopkins 
University/APL 

•  9052  -  Best  Practices  in  Contracting  for  Models,  Simulations,  and  Associated  Data,  Mr.  Dennis  Shea,  CNA 

•  8947  -  Report  on  a  Study  on  Management  Concepts  for  Broadly-Needed  Modeling  and  Simulation  Tools  in  the  US.  Department  of  Defense,  Dr. 
James  Coolahan,  Johns  Hopkins  University/APL 

•  8836  -  Producibility  Modeling  &  Simulation  Needs  for  Early  Systems  Engineering  Evaluations  of  Alternative  Design  Concepts,  Dr.  A1  Sanders, 
Honeywell  Aerospace 

•  8810  -  Using  Simulation  to  Define  and  allocate  probabilistic  Requirements,  Ms.  Yvonne  Bijan,  Lockheed  Martin  Aeronautics 

•  8923  -  Integration  of  Operational  Simulations  With  Physics-Based  Models  For  Engineering  Analysis,  Mr.  Stephen  Guest,  Lockheed  Martin 
Aeronautics 

TRACK  4  -  Practical  Systems  Engineering  -  Mission  I 

•  8980  -  Using  Model-driven  Engineering  Techniques  for  Integrated  Flight  Simulation  Development,  Mr.  Douglas  Fiehler,  Raytheon  Missile 
Systems 

•  9007  -  Technology  Maturation  for  the  Automated  Aerial  Refueling  (AAR)  Project,  Ms.  Carol  Ventresca,  SynGenics  Corporation 

•  8880  -  Naval  Postgraduate  School  Advanced  Seabase  Enabler  Project:  A  Systems  Engineering  Case  Study,  Mr.  Lance  Flitter,  NSWC, 

Carderock  Division 

•  8946  -  Protecting  the  Mission,  Preserving  Legacy  and  Promoting  Growth,  Ms.  Patti  Scaramuzzo,  Lockheed  Martin  Corporation 

•  9054  -  A-10  Avionics  System  Architecture  Trade  Study  and  Analysis  (A  VSATA)  Program,  Mr.  Richard  Sorensen,  KIHO  Military  Acquisition 

Consulting,  Inc. 

•  8976  -  A  Systems  Engineering  Model  for  Roadmap  Alignment,  Mr.  Si  Dok,  U.  S.  Army  TARDEC 

•  9080  -  Rapid  Systems  Engineering  of  the  MRAP  Gunner  Restraint  System  Saves  Lives,  Ms.  Michelle  Bowen,  JPO  MRAP 

•  9002  -  Key  Considerations  for  Building  Highly  Available,  Mission-Critical  Systems,  Mr.  Stephen  Mills,  GoAhead  Software 

TRACK  5  -  Human  Systems  Integration  -  Mission  II 

•  8937  -  Integrating  the  Human  into  the  system,  integrating  HSI  Tools  into  Systems  Engineering,  Dr.  Jennifer  Narkevicius,  Jenius  LLC 

•  9064  -  Economics  of  Human  Systems  Integration:  Early  Life  Cycle  Cost  Estimation  Using  HSI  Requirements,  2ndLt  Kevin  Liu,  USMC,  MIT 

•  Proccess  Management  and  tool  selection  to  minimize  risk  of  hand-arm  vibration  syndrome ,  Mr.  Sherman  Forbes,  U.S.  Air  Force 

TRACK  5  -  Systems  Engineering  Development  Environment  -  Mission  II 

•  8945  -  Standards  Based  Development  Environment,  Mr.  Christopher  Oster,  Lockheed  Martin  Corporation 

•  8922  -  The  Role  of  DoD  in  Systems  Engineering  Standards  and  Models,  Mr.  Donald  Gantzer,  Systems  Engineering  Directorate,  ODDR&E 

•  8844  -  The  Power  of  the  Spec:  Understanding  the  Many  Diverse  Roles  in  SE  of  Good  Specifications  &  Standards.  ”,  Mr.  Robert  Kuhnen,  U.S. 
Air  Force 
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•  8967  -  Generating  Visual  and  Interactive  Output  from  System  Engineering  Tools,  Mr.  John  Schatz,  Systems  and  Proposal  Engineering  Company 

•  9015  -  Challenges  and  Benefits  of  applying  ISO  STEP,  Mr.  Stuart  Booth,  Systems  Engineering  Directorate,  ODDR&E 

•  9059  -  Smalls  at  Conceptual  Design  Trade  and  Cost  Modeling  Tool,  Dr.  Deganit  Armon,  Advatech  Pacific,  Inc 

TRACK  6  -  Enterprise  Health  Management  -  Mission  III 

•  8815  -  Applying  Systems  Engineering  to  Operational  System  Improvements,  Ms.  Ryanne  Gentry,  Acquisition  Logistics  Engineering 

•  8842  -  Applications  in  Integrated  Diagnostics,  Mr.  Jimmy  Simmons,  Georgia  Tech  Research  Institute 

•  8884  -  Tactical  Wheeled  Vehicle  Integrated  Diagnostics ,  Mr  Lawrence  Osentoski,  DRIVE  Developments,  Inc. 

TRACK  6  -  System  of  Systems  -  Mission  III 

•  8964  -  Software  Assurance  in  a  System  of  Systems  World:  Interoperability  Challenges  -  Reports  from  the  Field,  Dr.  Carol  Sledge,  SEI 

•  8969  -  An  Introduction  to  Influence  Maps:  Foundations,  Construction,  and  Use,  Mr.  James  Smith,  SEI 

•  9024  -  Dynamic  Modeling  of  Programmatic  and  Systematic,  Dr.  Brian  Sauser,  Purdue  University 

•  8915  -  System  of  Systems  Challenges  and  Solutions:  Case  Study  Insights,  Mr.  John  Colombi,  U.S.  Air  Force  Institute  of  Technology 

TRACK  7  -  Work  Force  Development  -  Palm  I 

•  8966  -  Improving  Systems  Engineering  Curriculum  Using  a  Competency-Based  Assessment  Approach,  Ms.  Alice  Squires,  Stevens  Institute  of 
Technology 

•  9088  -  Enhancing  Systems  Engineering  Competencies  in  the  Enterprise,  Mr.  Gary  Roedler,  Lockheed  Martin  Corporation 

•  8789  -  Achieving  Acquisition  Excellence  via  Improving  the  Systems -Engineering  Workforce,  Dr.  Kenneth  Nidiffer,  SEI 

•  8926  -  Systems  Engineering  Workforce  Development  Update,  Dr.  Don  Gelosh,  Systems  Engineering  Directorate,  ODDR&E 

•  9076  -  Assessing  Systems  Engineering  Personnel  Competency:  Framework  and  Tool  Experience,  Dr.  Barry  Boehm,  University  of  Southern 
California 

•  8943  -  Team  SE  Skill  Set,  Mr.  Charles  Garland,  U.S.  Air  Force  Center  for  Systems  Engineering 

•  8956  -  Systems  Engineering  Approach  to  Workforce  Development,  Mr.  James  Miller,  U.S.  Air  Force 

•  9046  -  Developing  an  Introductory  Systems  Engineering  Practitioners  Course:  “Model-Based  Systems  Engineering  (MBSE)  With  SysML  ”,  Mr. 
Joseph  Wolfrom,  Johns  Hopkins  University/ APL 

•  8878  -  Advanced  Simulation  Course  for  Army  Simulation  Management  Professionals,  Dr.  Gene  Paulo,  Naval  Postgraduate  School 

TRACK  8  -  Software  Intensive  Systems  -  Palm  II 

•  8977  -  Overview  of  DoD  Software  Engineering  Initiatives,  Mr.  Scott  Lucero,  Systems  Engineering  Directorate,  ODDR&E 

•  8820  -  Graduate  Software  Engineering  Reference  Curriculum  (GSwERC),  Ms.  Nicole  Hutchison,  Analytic  Services,  Inc. 

•  8739  -  Quality  Assessment  of  Software-Intensive  System  Architectures  and  their  Requirements  (QUASAR),  Mr.  Donald  Firesmith,  SEI 

•  8812  -  ^4  Systems  Engineering  Approach  to  Multi-Level  Security  in  a  Service  Oriented  Architecture,  Mr.  Timothy  Greer,  Lockheed  Martin 
Corporation 

•  9104  -  Static  Code  Analysis:  Best  Practices  for  Software  Assurance  in  the  Acquisition  Life  Cycle,  Mr.  Paul  Croll,  CSC 

•  8996  -  Engineering  Improvement  in  Software  Assurance:  A  Landscape  Framework,  Ms.  Lisa  Brownsword,  SEI 

•  8802  -  Open  Source  Technology  for  Enterprise  Health  Management,  Mr.  Edward  Beck,  CSC 

•  8901  -  Review  Results  of  the  NDIA/OSD  Software  Test  Summit/Workshop,  Mr.  Thomas  Wissink,  Lockheed  Martin  IS&GS 

•  9506  -  Software  Acquisition  Management  Practical  Experience,  Mr.  James  Jones,  SSAI 
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SYSTEMS  ENGINEERING  CONFERENCE 
CONFERENCE  OVERVIEW 


►  CONFERENCE  OVERVIEW 

The  NDIA  Systems  Engineering  conference  is  focused  on  improving  acquisition  and  performance  of  Defense 
programs  and  systems,  including  net-centric  operations  and  data/information  interoperability,  system-of- 
systems  engineering  and  all  aspects  of  system  sustainment.  Convened  in  San  Diego,  CA,  October  26-29,  2009, 
this  conference  is  sponsored  by  the  National  Defense  Industrial  Association,  Systems  Engineering  Division, 
with  technical  co-sponsorship  by  IEEE  AES,  IEEE  Systems  Council  and  the  International  Council  on  Systems 
Engineering,  and  is  supported  by  the  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics,  Office  of  the  Director,  Defense  Research  &  Engineering/ Systems  Engineering. 

►  BACKGROUND 

The  Department  of  Defense  continues  to  work  to  improve  the  acquisition  of  military  equipment  and  capability 
to  assist  the  warfighter  in  protecting  the  U.S.  and  its  allies,  and  help  oppressed  nations  around  the  world,  amidst 
continuously  changing  conditions  and  threats.  The  DoD  seeks  to  improve  the  acquisition  process  and  overall 
program  execution  of  military  systems,  to  provide  greater,  more  effective  and  reliable  warfighting  capability,  at 
affordable  cost  and  within  reasonable  schedules.  One  of  the  primary  and  critically  important  areas  of  program 
acquisition  and  execution  lies  in  the  umbrella  discipline  of  Systems  Engineering,  which  is  the  overall  integrating 
function  in  defense  programs,  from  proper  requirements  definition  &  flowdown,  effective  and  affordable  design 
that  integrates  reliability,  availability  and  maintainability  considerations  into  the  overall  balance  of  design 
that  emphasizes  supportability  and  usage  aspects  along  with  overall  performance,  cost  and  schedule.  Systems 
Engineering  principles  embody  strong  technical  and  risk  management  aspects,  for  both  the  acquiring  program 
office  as  well  as  the  executing  defense  prime  and  subcontractors.  Strong  emphasis  on  Systems  Engineering 
throughout  the  life  cycle  of  the  program,  from  concept  development  through  sustainment,  is  a  key  enabler  of 
successful  programs.  The  annual  Systems  Engineering  Conference  explores  the  role  of  Systems  Engineering  in 
defense  programs  from  all  aspects  and  perspectives,  including  the  pragmatic,  practical  and  academic  viewpoints, 
and  brings  key  practitioners  together  to  work  on  effective  solutions  to  achieving  a  successful  warfighting  force. 

►  CONFERENCE  OBJECTIVES 

This  conference  seeks  to  create  an  interactive  forum  for  Program  Managers,  Systems  Engineers,  Chief  Scientists 
and  Engineers  and  Managers  from  the  Requirements,  Design,  Verification,  Support,  Logistics  and  Test 
communities  from  Government,  Academia,  and  Industry.  The  conference  will  provide  the  opportunity  to 
shape  policy  and  procedures  by  exchanging  innovative  tactics  and  lessons  learned. 


SYSTEMS  ENGINEERING  CONFERENCE 
ATTENDEE  INFORMATION 


►  CONTACTS 

Technical  Program 
Co-Chairs: 

Mr.  Steve  Henry, 

Manager,  Systems  Engineering 
and  Program  Support, 
Northrop  Grumman 
Information  Systems, 
stephen.henry@ngc.com, 

(703)  361-3724 


►  ATTIRE 

Appropriate  dress  for  this  conference  is  business  casual  for  civilians  and  class 
B  uniform  for  military.  During  conference  registration  and  check-in,  each 
participant  will  be  issued  an  identification  badge.  Please  be  prepared  to 
present  a  picture  ID.  Badges  must  be  worn  at  all  conference  functions. 

►  CONFERENCE  PROCEEDINGS 

Proceedings  will  be  available  on  the  web  through  the  Defense  Technical 
Information  Center  (DTIC),  and  will  be  available  one  to  two  weeks  after 
the  conference.  You  will  receive  notification  via  e-mail  once  proceedings  are 
posted  and  available  on  the  web. 


Dr.  Tom  Christian, 

ASC/EN, 

thomas.christian@wpafb.af.mil, 

(478)  926-2457 

Conference  Chair: 

Mr.  Bob  Rassa, 

Director,  Systems 
Supportability,  Raytheon; 
Chair,  Systems  Engineering 
Division,  NDIA, 
rcrassa@raytheon.com, 

(310)  985-4962 

Meeting  Planner: 

Ms.  Suzanne  Havelis, 

NDIA,  shavelis@ndia.org, 
(703)  247-2570. 

Conference  Director: 

Mr.  Sam  Campagna, 

NDIA,  scampagna@ndia.org, 
(703)  247-2544 


►  CONTINUING  EDUCATION  UNIT  CREDIT 

NDIA  is  offering  CEU  credit  options  for  the  Systems  Engineering 
Conference.  For  more  information,  please  contact  Ms.  Suzanne  Havelis  at 
703.247.2570  or  shavelis@ndia.org. 

►  2010  CALL  FOR  PAPERS  INFORMATION 

The  primary  objective  of  the  13th  Annual  Systems  Engineering  Conference 
is  to  provide  insight,  information  and  lessons  learned  into  how  we  can 
improve  the  overall  performance  of  defense  programs  via  a  better,  more 
focused  application  of  systems  engineering  that  will  lead  to  more  capable, 
interoperable  and  supportable  weapon  systems  for  the  warfighter,  with 
reduced  total  ownership  costs,  to  help  our  military  meet  its  current  and  new 
mission  area  and  capabilities  requirements.  Technical  and  management 
presentations  are  a  key  tactic  in  achieving  this  objective.  You  are  invited  to 
submit  a  short  (under  300  word)  abstract  of  a  presentation  for  a  session  (see 
topics  on  the  website).  Abstracts  must  fully  describe  the  planned  content 
and  how  the  presentations  will  advance  the  objectives  of  the  conference  and 
session.  All  accepted  presentations  will  be  delivered  at  the  conference  in 
electronic  format;  full  papers  are  optional  and  are  not  required. 

Abstracts  must  include  the  following  administrative  information: 
presentation  title,  authors  name,  title,  e-mail  address,  phone  number, 
mailing  address  and  organization  and  the  conference  session  targeted. 
Abstracts  must  be  submitted  no  later  than  Sunday,  May  30,  2010  via  the 
following  web  link: 

http://application.ndia.org/abstracts/1870 

Abstracts  will  only  be  accepted  through  this  web  link,  and  all  required 

information  must  be  completed.  Upon  completion 

of  the  required  information,  you  will  receive  an  e-mail  confirmation. 

**Conference  presenters  are  not  exempt  from  registration  and  conference 
fees. 
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OVERVIEW:  OCTOBER  25-0CT0BER  27 

CONFERENCE  AGENDA 

SUNDAY,  OCTOBER  25,  2009 

5:00  pm  -  7:00  pm  REGISTRATION  FOR  TUTORIALS  AND  GENERAL  CONFERENCE 


MONDAY,  OCTOBER  26,  2009 


7:00  am  -  6:00  pm 
7:00  am  -  8:00  am 
8:00  am  -  12:00  pm 
9:45  am  -  10:15  am 
12:00  pm  -  1:00  pm 
1:00  pm -5:00  pm 
2:45  pm  -  3:15  pm 
5:00  pm  -  6:00  pm 


REGISTRATION 

CONTINENTAL  BREAKFAST  (FOR  TUTORIAL  ATTENDEES  ONLY) 

TUTORIAL  TRACKS 

MORNING  BREAK  (FOR  TUTORIAL  ATTENDEES  ONLY) 

LUNCH  (FOR  TUTORIAL  ATTENDEES  ONLY) 

TUTORIAL  TRACKS  CONTINUED 

AFTERNOON  BREAK  (FOR  TUTORIAL  ATTENDEES  ONLY) 

RECEPTION  IN  THE  REGATTA  PAVILION  -  OPEN  TO  ALL  CONFERENCE  ATTENDEES 


TUESDAY,  OCTOBER  27,  2009 


7:15  am  -  7:00  pm 
7:15  am  -  8:15  am 
8:15  am  -  8:30  am 


REGISTRATION 

CONTINENTAL  BREAKFAST  IN  THE  REGATTA  PAVILION 
PLENARY  SESSION  1  -  INTRODUCTION  &  OPENING  REMARKS 


►  Mr.  Sam  Campagna,  Director ;  Operations ,  NDIA 

►  Mr.  Bob  Rassa,  Director ;  Systems  Supportability ;  Raytheon ;  Chair, 
Systems  Engineering  Division,  NDIA 


8:30  am  -  9:30  am  KEYNOTE 

►  Honorable  Zachary  J.  Lemnios,  Director,  Defense  Research  and  Engineering,  Office  of  the 
Under  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics ) 


9:30  am  -  10:00  am  MORNING  BREAK  IN  THE  REGATTA  PAVILION 

10:00  am  -  12:00  pm  PLENARY  SESSION  2  -  ACQUISITION  EXECUTIVES  PANEL 

View  from  the  Top:  How  Can  SE  Support  Program  Execution ? 

Moderator:  Mr.  Terry  Jaggers,  Principal  Deputy,  Systems  Engineering,  Office  of  the  Director, 
Defense  Research  and  Engineering 

►  Mr.  David  G.  Ahern,  Director,  Por folio  Systems  Acquisition,  Office  of  the  Under  Secretary 
of  Defense  (Acquisition,  Technology  and  Logistics ) 

►  Mr.  Thomas  E.  Mullins,  Deputy  Assistant  Secretary  for  Plans,  Programs,  and  Resources 
(SAAL-ZR),  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics  and 
Technology) 

►  Mr.  Christopher  A.  Miller,  PEO  for  Command,  Control,  Communications,  Computers 
and  Intelligence  (C4I),  US.  Navy 

►  Mr.  Randall  G.  Walden,  Director,  Information  Dominance  Programs,  Office  of  the 
Assistant  Secretary  of  the  Air  Force  (Acquisition) 


12:00  pm  -  1:30  pm 


LUNCH  WITH  SPEAKER  IN  THE  REGATTA  PAVILION 

►  Mr.  Stephen  Welby,  Director,  Systems  Engineering,  Office  of  the  Director,  Defense 
Research  and  Engineering 


SYSTEMS  ENGINEERING  CONFERENCE 

OVERVIEW:  OCTOBER  27-0CT0BER  29 

TUESDAY,  OCTOBER  27,  2009  -  CONTINUED 


1:30  pm  -  3:15  pm 


3:15  pm -3:30  pm 
3:30  pm  -  5:15  pm 


PLENARY  SESSION  3  -  TEST  &  EVALUATION  EXECUTIVES  PANEL 

View  from  the  Top:  How  SE  Can  Support  Test  and  Evaluation ? 

Moderator:  Mr.  Jim  O’Bryon,  The  O’Bryon  Group;  Chair,  NDIA  Test  and  Evaluation  Division 

►  Dr.  James  N.  Streilein,  Technical  Advisor,  HQ  Army  Test  &  Evaluation  Command 

►  Ms.  Amy  Markowich,  Deputy  DoN  T&E  Executive 

►  Colonel  Dexter  M.  Sapinoso,  USAF,  Chief  of  Air  Force  Test  and  Evaluation  Policy  and 
Programs 

►  Mr.  Christopher  DiPetto,  Acting  Director,  DevelopmentalTest  and  Evaluation,  Office  of  the 
Director,  Defense  Research  and  Engineering 

AFTERNOON  BREAK  IN  THE  REGATTA  PAVILION 

PLENARY  SESSION  4  -  SE  AND  ACQUISITION  REFORM:  THE  WAY  AHEAD 

Moderator:  Mrs.  Kristen  Baldwin,  Systems  Engineering  Director  ate,  Off ce  of the  Director, 
Defense  Research  and  Engineering 

►  Mr.  Ross  Guckert,  Office  of  the  Secretary  of  the  Army  for  Acquisition,  Logistics  and 
Technology  (ASA(ALT)) 

►  Mr.  Carl  Siel,  Office  of  the  Assistant  Secretary  of  the  Navy  for  Research,  Development  and 
Acquisition  (ASNfRDA )  CHSENG) 

►  Colonel  Shawn  Shanley,  USAF,  Chief  Systems  Engineer,  Office  of  the  Assistant  Secretary  of  the 
Air  Force  for  Acquisition,  Science,  Technology,  and  Engineering  (SAF/AQR) 

►  Mr.  Nicholas  Torelli,  Systems  Engineering  Directorate,  Office  of  the  Director,  Defense  Research 
and  Engineering 


5:30  pm  -  7:00  pm  RECEPTION  IN  THE  REGATTA  PAVILION 

WEDNESDAY,  OCTOBER  28,  2009 


7:00  am  -  5:15  pm 
7:00  am  -  8:00  am 
8:00  am  -  12:00  pm 
9:45  am  -  10:15  am 
12:00  pm  -  1:30  pm 
1:30  pm  -  5:15  pm 
3:15  pm -3:30  pm 
5:15  pm 


REGISTRATION 

CONTINENTAL  BREAKFAST  IN  THE  REGATTA  PAVILION 

CONCURRENT  SESSIONS  -  Please  refer  to  the  following  pages  for  session  schedule 

MORNING  BREAK  IN  THE  REGATTA  PAVILION 
AWARDS  LUNCH  IN  THE  REGATTA  PAVILION 

CONCURRENT  SESSIONS  -  Please  refer  to  the  following  pages  for  session  schedule 

AFTERNOON  BREAK  IN  THE  REGATTA  PAVILION 
WEDNESDAY  SESSION  ADJOURNS 


THURSDAY,  OCTOBER  29,  2009 


7:00  am  -  3:00  pm 
7:00  am  -  8:00  am 
8:00  am  -  12:00  pm 
9:45  am  -  10:15  am 
12:00  pm  -  1:00  pm 
1:00  pm  -  3:00  pm 


REGISTRATION 

CONTINENTAL  BREAKFAST  IN  THE  REGATTA  PAVILION 

CONCURRENT  SESSIONS  -  Please  refer  to  the  following  pages  for  session  schedule 

MORNING  BREAK  IN  THE  REGATTA  PAVILION 
LUNCH  IN  THE  REGATTA  PAVILION 

CONCURRENT  SESSIONS  -  Please  refer  to  the  following  pages  for  session  schedule 


3:00  pm 


CONFERENCE  ADJOURNS 


MONDAY,  OCTOBER  26,  TUTORIAL  SESSIONS 


TRACK 

8:00  AM 
SESSION  A 

10:15  AM 
SESSION  B 

1:00  PM 
SESSION  C 

3:15  PM 
SESSION  D 

TRACK  8 

Palm  II 

8819  -  1A8  -  Tutorial: 

Rethinking  Risk  Management 

Ms.  Audrey  Dorofee,  SEI/ 

CMU 

8819  -  1B8  -  Tutorial: 

Rethinking  Risk  Management 

Ms.  Audrey  Dorofee,  SEI/CMU 

8877  -  1C8  -  Tutorial:  Best 
Practices  in  Modeling  and 
Simulation 

Dr.  Gene  Paulo,  Naval 
Postgraduate  School 

8877-  1D8 -Tutorial:  Best 
Practices  in  Modeling  and 
Simulation 

Dr.  Gene  Paulo,  Naval 
Postgraduate  School 

TRACK 7 

Palm  I 

8785  -  1A7  -  Tutorial:  Agile 
Development  in  Defense 
Acquisition 

8785  -  1B7  -  Tutorial:  Agile 
Development  in  Defense 
Acquisition 

8801  -  1C7- Tutorial: 

Integrating  SE  with  Earned 

Value  Management 

8801  -  1C7- Tutorial: 

Integrating  SE  with  Earned 

Value  Management 

Dr.  Peter  Hantos,  The 

Aerospace  Corporation 

Dr.  Peter  Hantos,  The  Aerospace 
Corporation 

Mr.  Paul  Soloman,  Performance- 
Based  Earned  Value 

Mr.  Paul  Soloman,  Performance- 
Based  Earned  Value 

v©  S 

y  .2 

9078  -  1A6  -  Tutorial: 
Organizational  Implications 
of  SoS 

9078  -  1B6  -  Tutorial: 
Organizational  Implications 
of  SoS 

8782  -  1C6  -  Tutorial: 

Technology  Transition  and  the 
Defense  Acquisition  System 

8782  -  1C6  -  Tutorial: 

Technology  Transition  and  the 
Defense  Acquisition  System 

H  S 

Ms.  Suzanne  Garcia,  SEI/CMU 

Ms.  Suzanne  Garcia,  SEI/CMU 

Mr.  William  Decker,  DAU 

Mr.  William  Decker,  DAU 

TRACK  5 

Mission  II 

8984  -  1A5  -  Tutorial:  How  to 
use  Lean  SE  Processes  to  Save 
Time  and  Money 

Mr.  Tim  Olson,  Lean  Solutions 
Institute,  Inc. 

8984-  1B5  -  Tutorial:  How  to 
use  Lean  SE  Processes  to  Save 
Time  and  Money 

Mr.  Tim  Olson,  Lean  Solutions 
Institute,  Inc. 

9072  -  1C5  -  Tutorial: 

Leveraging  the  Defense  Acq 
Program  Support  (DAPS) 
Methodology  to  Conduct 
Program  Assessment 

Mr.  Peter  Nolte,  Systems 
Engineering  Directorate, 
ODDR&E 

9072  -  1D5  -  Tutorial: 

Leveraging  the  Defense  Acq 
Program  Support  (DAPS) 
Methodology  to  Conduct 

Program  Assessment 

Mr.  Peter  Nolte,  Systems 
Engineering  Directorate, 
ODDR&E 

s«tl  i— i 

bl 

9035  -  1A4  -  Tutorial: 
Collaborative  Decision  Making 

9035  -  1B4  -  Tutorial: 
Collaborative  Decision  Making 

8931  -  1C4  -  Tutorial:  Role  of 
Mentoring  in  Developing  the 

Sys  Eng  Workforce 

8931  -  1D4  -  Tutorial:  Role  of 
Mentoring  in  Developing  the 

Sys  Eng  Workforce 

gi 

Dr.  Tommer  Ender,  Georgia 

Tech  Research  Institute 

Dr.  Tommer  Ender,  Georgia 

Tech  Research  Institute 

Mr.  Nicholas  Torelli,  Systems 
Engineering  Directorate, 
ODDR&E 

Mr.  Nicholas  Torelli,  Systems 
Engineering  Directorate, 
ODDR&E 

TRACK 3 

Bayview  I 

8955  -  1A3  -Tutorial:  Early 

Sys  Thinking  and  Planning  in 
WPN  Sys  Concept  Phase 

Mr.  Jeff  Loren,  SAF/AQR 
(Alion  Science  &  Technology) 

8955  -  1B3  -Tutorial:  Early  Sys 
Thinking  and  Planning  in  WPN 
Sys  Concept  Phase 

Mr.  Jeff  Loren,  SAF/AQR 
(Alion  Science  &  Technology) 

9040  -  1C3  -  Tutorial: 
Implementing  the  Materiel 
Availability  KPP  in  DoD 
Acquisition  Programs 

Mr.  Grant  Schmieder,  Systems 
Engineering  Directorate, 
ODDR&E 

9040  -  1D3  -  Tutorial: 
Implementing  the  Materiel 
Availability  KPP  in  DoD 
Acquisition  Programs 

Mr.  Grant  Schmieder,  Systems 
Engineering  Directorate, 
ODDR&E 

rs  a 

u  § 

8779  -  1A2  -  Tutorial:  Mission 
Based  Test  and  Eval  Strategy: 
Case  Study 

8779  -  1B2  -  Tutorial:  Mission 
Based  Test  and  Eval  Strategy: 

Case  Study 

8818-  1C2- Tutorial: 

Integrated  Testing  Enhances  SE 

8818  -  1D2  -  Tutorial: 

Integrated  Testing  Enhances  SE 

TRA 

Bayv 

Mr.  Christopher  Wilcox,  U.S. 
Army  Test  and  Evaluation 
Command 

Mr.  Christopher  Wilcox,  U.S. 
Army  Test  and  Evaluation 
Command 

Dr.  Beth  Wilson,  Raytheon 
Company 

Dr.  Beth  Wilson,  Raytheon 
Company 

TRACK  1 
Bayview  III 

8736  -1A1  -  Tutorial: 

Framework  of  Engineering 
Architectures 

8736-  1B1  -  Tutorial: 

Framework  of  Engineering 
Architectures 

8992  -1C1  -Tutorial:  SoS 

Quality  Attribute  Specification 
and  Architecture  Evaluation 

8992  -1D1  -Tutorial:  SoS 

Quality  Attribute  Specification 
and  Architecture  Evaluation 

Mr.  Donald  Firesmith,  SEI 

Mr.  Donald  Firesmith,  SEI 

Mr.  Michael  Gagliardi,  SEI 

Mr.  Michael  Gagliardi,  SEI 

WEDNESDAY,  OCTOBER  28,  CONCURRENT  SESSIONS 


11:25  AM 

8913  -  Linking 

Interoperability 

and  Measures  of 

Effectiveness:  A 

Method  for  Evaluating 

Architectures 

Dr.  David  Jacques, 

Air  Force  Institute  of 

Technology 

8895  -  A  Comprehensive 

Review  of  Maturity 

Assessment  Approaches 

for  Improved  Defense 

Acquisition 

Ms.  Nazanin  Azizian, 

The  George  Washington 

University 

8961  -  Engineering 

Systems  of  Systems:  An 

Integration  Perspective 

Dr.  Emmett  Maddry, 

NSWCDD 

9091  -  Environment, 

Safety,  and  Occupational 

Health  (ESOH) 

Risk  and  Technology 

Requirements  Reporting 

at  Acquisition  Program 

Reviews 

Ms.  Lucy  Rodriguez, 

Booz  Allen  Hamilton 

10:50  AM 

9081  -  Testing  in 
Service-oriented 
Environments 

Mr.  Soumya  Simanta, 

SEI 

8791  -  Cost  and  Risk 

Impacts  of  the  New 

DOD  5000  Defense 

Acquisition  Framework 

Dr.  Peter  Hantos,  The 

Aerospace  Corporation 

8942  -  DoD  Systems  of 

Systems  Update 

Dr.  Judith  Dahmann, 

Systems  Engineering 

Directorate,  ODDR&E/ 

MITRE 

9094  -  DoD  Green 

Procurement  Program 

Update  and  Path 

Forward 

Mr.  David  Asiello, 

Office  of  the  Secretary  of 

Defense 

10:15  AM 

8929  -  Extending 
Net-Centric  Quality  of 
Service  to  Systems  of 
Systems 

Maj  Vinod  Naga,  USAF, 
Air  Force  Institute  of 
Technology 

8982  -  Systemic  Root 
Cause  Analysis  -  Driving 
Improvements  into  the 
Acquisition  Process 

Mr.  Peter  Nolte,  Systems 

Engineering  Directorate, 

ODDR&EE) 

8840  -  Naval  Systems 

of  Systems  Engineering 

Guidebook  Update 

Ms.  Melinda  Reed,  DoD 

(ASN  RDA  CHSENG) 

9070  -  Improving  Safety 

Technology  Insertion 

in  DoD  Acquisition 

Programs 

Dr.  Elizabeth  Rodriguez- 

Johnson,  Systems 

Engineering  Directorate, 

ODDR&E 

SESSION 

CHAIR 
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9:10  AM 

8853  -  C4I  Architecture 
for  Joint  AS  W 

Mr.  Gregory  Miller, 

Naval  Postgraduate 

School 

9065  -  Rapidly 
Implementing  Lean 
CMMI®  Processes  That 
Meet  Business  Needs 

Mr.  Tim  Olson,  Lean 
Solutions  Institute,  Inc 

8784  -  Establishing 
a  Departmental- 
Level  Systems-of- 
Systems  Engineering 
Management  Construct 
for  the  Department 
of  the  Navy,  Progress 
Report 

Mr.  John  Kevin  Smith, 
Asst  Sec  of  the  Navy  for 
RD&A,  Chief  Engineer 

8829  "  The  Army 

Health  Hazard 

Assessment  Program’s 

Medical  Cost  Avoidance 

Model 

Dr.  Timothy  A. 

Kluchinsky,  Department 

of  Army 

8:35  AM 

8788  -  Data  sharing  in 
a  Stability  Operations 
Community  of  Interest: 
Utilizing  a  pilot  program 
to  prove  concepts  and 
develop  trust. 

Mr.  Gerald  Christman, 
Femme  Comp  Inc. 

9034  -  Sustainment 
and  Continued 
Institutionalization 
of  Best  Practices  and 
CMMI®  at  SPAWAR 

Mr.  Michael  Kutch, 
SPAWAR  Systems 

Center  Atlantic 

8960  -  A  Distillation  of 
Lessons  Learned  from 
Complex  System  of 
Systems  Acquisitions 

Dr.  Richard  Turner, 

Stevens  Institute 

9042  -  Bounding  the 
Human  Within  the 

System 

Mr.  Michael  Mueller, 

U.S.  Air  Force  Center 

for  Systems  Engineering 

8:00  AM 

8780  -  Net-Centric  Best 

Practices 

Mr.  Hiekeun  Ko,  JPEO- 
CBD  -  Software  Support 
Activity 

9003  -  CMMI®  for 
Executives 

Mr.  Geoff  Draper,  Harris 
Corporation 

NDIA  SoS  Committee 
Report 

Dr.  Judith  Dahman, 
Systems  Engineering 
Directorate,  ODDR&E/ 
MITRE 

8975  -  What  is  Human 
Systems  Integration 
(HSI)  and  why  should 
we  do  it? 

Mr.  Stuart  Booth, 

Systems  Engineering 
Directorate,  ODDR&E 

SESSION 

CHAIR 
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8891  "A  comprehensive 

overview  of  techniques 

for  measuring  system 

readiness 

Mr.  James  Bilbro,  JB 

Consulting  International 

9026  -  Early  SE 

Determination  of  Best- 

Fit  System  Life  Cycle 

Processes 

Dr.  Barry  Boehm,  USC 

8839  -  Navy  Systems 

Engineering  Technical 

Review  Process 

Ms.  Melinda  Reed,  DoD 

(ASN  RDA  CHSENG) 

8901  -  Review  Results 

of  the  NDIA/OSD 

Software  Test  Summit/ 

Workshop 

Mr.  Thomas  Wissink, 

Lockheed  Martin 

IS&GS 

8833  "  Communicating 
Risk:  Air  Force  RI3 
Methodology 

Mr.  John  Cargill,  AF 

Cost  Analysis  Agency 

8813  -  Emerging  Roles 

for  Systems  Engineering 

in  Defense  Decision 

Making;  Better  Aligning 

Requirements  and 

Acquisition  with  the 

Budget  and  Security 

Environments 

Mr.  Vincent  Roske, 

Institute  for  Defense 

Analyses 

8823  -  Win  and 

Influence  Design 

Engineers — Change 

Their  Affordability  DNA 

Mr.  Tim  Morrill, 

Raytheon  Company 

8814  -  Joint  Mission 
Environment  Test 
Capability  (JMETC), 
Lowering  technical 

Risk  by  Improving 
Distributed  Test 
Capabilities 

Mr.  Chip  Ferguson, 
JMETC 

8894  -  Air  Force 

Initiative  —  High 
Confidence  Technology 
Transition  Planning 
Through  the  Use  of 
Stage-Gates  —  Update 

Mr.  Randy  Bullard, 

U.S.  Air  Force  Materiel 
Command 

8949  -  Updated  DoD 
5000  and  CJCS  3170 
Policies:  A  Requirements 

to  Acquisition  Gap 

Analysis 

Mr.  John  Lohse, 

Raytheon  Company 

8863  -  Using 

Requirements 

Compliance  to  Identify 

Gaps  Between  the 

Technical  Solution  and 

Requirements 

Mr.  Frank  Salvatore, 

High  Performance 

Technologies,  Inc. 
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8883  -  Test  & 
Evaluation  Products 
for  the  Systems 
Engineering  Reviews 

Mr.  Woody  Eischens, 

OUSD(AT&L)/ 

DDR&E/DT&E 

8900  -  DOD’s 

Weapon  System 
Portfolio:  Are  Results 
Getting  Any  Better? 

Mr.  Michael  Sullivan, 
U.S.  Government 
Accountability  Office 

Q&A:  8924,  8925 , 

8933-  Early  Systems 
Engineering  in  DoDI 
5000.02 

Dr.  Judith  Dahmann, 

Ms.  Lisa  Reuss, 

Systems  Engineering 
Directorate,  ODDR&E 

8974  -  Transforming 
Systems  and  Software 
Engineering  Across  an 
Enterprise 

Mr.  Jeffery  Wilcox, 
Lockheed  Martin 
Corporation 

8882  -  Test  &  Evaluation 
Strategy  for  the 

Technology  Development 
Phase 

Ms.  Darlene  Mosser- 
Kerner,  OUSD(AT&L)/ 
DDR&E/DT&E 

8963  "  Air  Force  Concept 
Maturity  Assessment 

Mr.  George  Freeman, 

U.S.  Air  Force,  Center  for 
Systems  Engineering 

Panel  Topic:  8924,  8925 , 8933  -  Early  Systems 

Engineering  in  DoDI  5000.02 

Dr.  Judith  Dahmann,  Ms.  Lisa  Reuss,  Systems 

Engineering  Directorate,  ODDR&E 

8990  -  Systems  Engineering 
for  Rapid  Capability 
Development 

Mr.  Thomas  McDermott, 
Georgia  Tech  Research 
Institute 

8848  -  Integrated 

Testing:  We  Can  Do  It 

Dr.  Beth  Wilson, 

Raytheon  Company 

8916  -  System  Readiness 
-  Assessing  Technical  Risk 
Throughout  the  Lifecycle 

Mr.  James  Thompson, 
Systems  Engineering 
Directorate,  ODDR&E 

8816  -  Mind  the  GAPs-a 
Systems  Engineering 
Implementation  of  DoDI 
5000.02 

Dr.  Thomas  Christian, 

U.  S.  Air  Force 
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8973  -  C- 17  Transition  to 

Criteria-based  Airworthiness 

Certification 

Mr.  Christian  Stillings,  USAF 

516AESG 

8879  -  TRL  Vectors  in  IPPD- 

based  Portfolio  Management 

Mr.  Michael  Bartmess, 

General  Dynamics/AIS 

8795  -  Mission-based  Test 

and  Evaluation  Strategy: 

Creating  Linkages  between 

Technology  Development  and 

Mission  Capability 

Mr.  John  Beilfuss,  U.S.  Army 

Research  Laboratory 

8948  -  Value  Engineering 

Applications  in  Service 

Contracts 

Dr.  Jay  Mandelbaum,  Value 

Engineering  Applications  in 

Service  Contracts 

8855  "  Certify  and  Fly 
Right:  Preparing  for  DO- 
297  Certification 

Mr.  Ketih  Custer, 

Esterline  Control 

Systems- AVISTA 

8870  -  S&T  Portfolio 
Maturity  &  Performance 
Analysis:  The  Concept 

of  Critical  Research 

Elements 

Mr.  Has  Patel,  Infologic, 

Inc. 

8835  -  T&E 

Collaboration  and 

Contributions  during 

Early  Program 

Acquisition 

Mr.  Stephen  Scukanec, 

Northrop  Grumman 

Corporation  Aerospace 

Systems 

9414  -  Correcting 

Deficiencies  in  the 

Systems  Engineering  of 

Tactical  Weapons 

Mr.  Marvin  Ebbert, 

Raytheon  Missile 

Systems 

9014  -  SAVI:  Aerospace 
Platform  Development 
and  Certification  Using 
Modeling  and  Simulation 
to  “Integrate,  then  Build” 

Mr.  Gregory  Pollari, 

Rockwell  Collins 

8798  -  The  New 
Technology  Readiness 

Assessment  Process 

Dr.  Jay  Mandelbaum, 

Institute  for  Defense 

Analyses 

9082  -  Including 
Environment,  Safety,  and 

Occupational  Health 

(ESOH)  Requirements 

in  Joint  Capabilities 

Integration  and 

Development  System 

(JCIDS)  Documents 

Mr.  Sherman  Forbes, 

U.S.  Air  Force 

8902  -  Systems 

Engineering  Leading 

Indicators:  Insight 

into  Effective  Systems 

Engineering 

Mr.  Gary  Roedler, 

Lockheed  Martin 

Corporation 
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8935  -  Systems  of 

Systems  Systems 
Engineering  and  Test  and 
Evaluation 

Dr.  Judith  Dahmann, 
Systems  Engineering 
Directorate,  ODDR&E/ 

MITRE 

8770  -  Incorporating 
Maturity  Assessment 
into  House  of  Quality 
for  Improved  Decision 
Support  Analysis  and 

Risk  Management 

Mr.  Pavel  Fomin,  U.S. 

Air  Force 

9016  -  A  Framework  for 
Enhancing  Forward- 
looking  Capability 
Delivery  Metrics 

Mr.  Leonard  Sadauskas, 

DoD  CIO  CT&S 

8847  -  Tailoring  the  SE 
Process  to  Effectively 
Complement  the  SW 

Agile  Development 

Process 

Mr.  William  Lyders, 

ASSETT  Inc. 

8849  -  Joint  Integration  and 
Interoperability  Lab  (JSIIL) 

Mr.  Steven  Whitehead, 

SL,  J8  Technical  Director, 
USJFCOM 

9017  -  Linking  Systems 
Engineering  Artifacts  with 
Complex  Systems  Maturity 

Assessments 

Dr.  Brian  Sauser,  Stevens 
Institute  of  Technology 

8846  -  Air  Force  Materiel 

Command  Early  Systems 
Engineering 

Dr.  Brian  Kowal,  USAF 

8893  -  Rapid  Development 

Mr.  Michael  Gaydar, 

NAVAIR 

8825  -  Test  and  Evaluation 
in  a  System  of  Systems 
Environment 

Mr.  Edwin  McDermott,  653 
ELSW,  Electronic  Systems 
Center 

899 1  -  Systems  Engineering 
for  the  Science  &  Technology 
Community 

Mr.  Russell  Menko,  U.S. 

Army  RDECOM/TARDEC 

8951  -  USAF  View  of  NRC 
“Pre-A  Systems  Engineering” 
Study  Committee 

Recommendations  As 

Addressed  By  Levin-McCain 
(P.L.  111-23;  “Weapon 

Systems  Acquisition  Reform 

Act  of  2009”) 

Mr.  Jeff  Loren,  SAF/AQR 
(Alion  Science  &  Technology) 

8851  -  Rapid  Development 
and  Integration  of  Remote 
Weapon  Systems  to  Meet 
Operational  Requirements 

Mr.  Joseph  Burkart,  NSWC 
Crane,  Small  Arms  Air 

Platform  Integration 
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THURSDAY,  OCTOBER  29,  CONCURRENT  SESSIONS 


11:25  AM 

8996  -  Engineering 

Improvement  in 

Software  Assurance:  A 

Landscape  Framework 

Ms.  Lisa  Brownsword, 

SEI 

8789  "  Achieving 

Acquisition  Excellence 

via  Improving  the 

Systems-Engineering 

Workforce 

Dr.  Kenneth  Nidiffer, 

SEI 

8915  -  System  of 

Systems  Challenges  and 

Solutions:  Case  Study 

Insights 

Mr.  John  Colombi, 

U.S.  Air  Force  Institute 

of  Technology 

8844  -  The  Power 

of  the  Spec: 

Understanding 

the  Many  Diverse 

Roles  in  SE  of  Good 

Specifications  & 

Standards.” 

Mr.  Robert  Kuhnen, 

U.S.  Air  Force 

10:50  AM 

9104  -  Static  Code 
Analysis:  Best  Practices 
for  Software  Assurance 

in  the  Acquisition  Life 

Cycle 

Mr.  Paul  Croll,  CSC 

9088  -  Enhancing 

Systems  Engineering 

Competencies  in  the 

Enterprise 

Mr.  Gary  Roedler, 

Lockheed  Martin 

Corporation 

8903  -  Global  Earth 

Observation  System  of 

Systems  (GEOSS) 

Mr.  Lawrence 

McGovern,  Northrop 

Grumman  Electronic 

Systems 

8922 -The  Role  of  DoD 

in  Systems  Engineering 

Standards  and  Models 

Mr.  Donald  Gantzer, 

Systems  Engineering 

Directorate,  ODDR&E 

10:15  AM 

8812  -  A  Systems 
Engineering  Approach 
to  Multi-Level  Security 
in  a  Service  Oriented 
Architecture 

Mr.  Timothy  Greer, 
Lockheed  Martin 
Corporation 

8966  -  Improving 

Systems  Engineering 

Curriculum  Using  a 

Competency-Based 

Assessment  Approach 

Ms.  Alice  Squires, 

Stevens  Institute  of 

Technology 

9024  -  Dynamic 

Modeling  of 

Programmatic  and 

Systematic 

Dr.  Brian  Sauser, 

Purdue  University 

8945  -  Standards 

Based  Development 

Environment 

Mr.  Christopher  Oster, 

Lockheed  Martin 

Corporation 
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9:10  AM 

8739  -  Quality  Assessment 
of  Software-Intensive 

System  Architectures 
and  their  Requirements 
(QUASAR) 

Mr.  Donald  Firesmith, 

SEI 

8943  -  Team  SE  Skill  Set 

Mr.  Charles  Garland, 

U.S.  Air  Force  Center  for 

Systems  Engineering 

8969  -  An  Introduction 
to  Influence  Maps: 
Foundations, 

Construction,  and  Use 

Mr.  James  Smith,  SEI 

Process  management  and 

tool  selection  to  minimize 

risk  of  hand-arm  vibration 

syndrome 

Mr.  Sherman  Forbes,  U.S. 

Air  Force 

8:35  AM 

8820  -  Graduate  Software 
Engineering  Reference 
Curriculum  (GSwERC) 

Ms.  Nicole  Hutchison, 
Analytic  Services,  Inc. 

9076  -  Assessing  Systems 
Engineering  Personnel 
Competency:  Framework 
and  Tool  Experience 

Dr.  Barry  Boehm, 

University  of  Southern 
California 

8964  -  Software  Assurance 
in  a  System  of  Systems 
World:  Interoperability 
Challenges  -  Reports  from 

the  Field 

Dr.  Carol  Sledge,  SEI 

9064  -  Economics 
of  Human  Systems 
Integration:  Early  Life 

Cycle  Cost  Estimation 

Using  HSI  Requirements 

2ndLt  Kevin  Liu,  USMC, 

MIT 

8:00  AM 

8977  _  Overview  of  DoD 
Software  Engineering 

Initiatives 

Mr.  Scott  Lucero,  Systems 
Engineering  Directorate, 

ODDR&E 

8926  -  Systems  Engineering 
Workforce  Development 
Update 

Dr.  Don  Gelosh,  Systems 
Engineering  Directorate, 

ODDR&E 

9083  -  Requirements 
Engineering  for  Systems  of 
Systems 

Mr.  Soumya  Simanta,  SEI 

8937  -  Integrating  the 

Human  into  the  system, 
integrating  HSI  Tools  into 
Systems  Engineering 

Dr.  Jennifer  Narkevicius, 
Jenius  LLC 
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9054  -  A- 10  Avionics 
System  Architecture 
Trade  Study  and 

Analysis  (AVSATA) 
Program 

Mr.  Richard 

Sorensen,  KIHO 

Military  Acquisition 

Consulting,  Inc. 

8947  _  Report  on  a 

Study  on  Management 

Concepts  for  Broadly- 

Needed  Modeling  and 

Simulation  Tools  in  the 

U.S.  Department  of 

Defense 

Dr.  James  Coolahan, 

Johns  Hopkins 

University/APL 

9008  -  Upgrade  Fluid 

System  Filter  Element 

Monitoring  To  Increase 

Operational  Reliability 

and  Support  Condition 

Based  Maintenance 

Capability 

Mr.  Gary  Rosenberg, 

Constellation 

Technology 

Corportation 

8824  -  Delivering 

DoDAF  Version  2.0  to 

Architects  and  Systems 

engineers  for  IT 

Systems  and  Services 

Mr.  Walt  Okon, 

Department  of  Defense 

CIO,  Enterprise 

Architecture 

8946  -  Protecting  the 
Mission,  Preserving 

Legacy  and  Promoting 

Growth 

Ms.  Patti  Scaramuzzo, 

Lockheed  Martin 

Corporation 

9052  -  Best  Practices 
in  Contracting  for 

Models,  Simulations,  and 

Associated  Data 

Mr.  Dennis  Shea,  CNA 

9039  -  Applying  Systems 

Engineering  to  Fielded 

Weapon  Systems  and 

End-Items 

Mr.  Michael  Ucchino, 

AF  Center  for  Systems 

Engineering 

8830  -  Systems 

Engineering  Needs  of  the 

DoDAF  -  Report  of  the 

Architecture  Frameworks 

Working  Group 

Mr.  Joe  Kuncel, 

Northrop  Grumman 

Corporation 

8880  -  Naval 

Postgraduate  School 
Advanced  Seabase 

Enabler  Project:  A 

Systems  Engineering 

Case  Study 

Mr.  Lance  Flitter, 

NSWC,  Carderock 

Division 

8759  -  A  Systems 
Engineering  Framework 
for  Integrating  M&S 
Development  Best 

Practices 

Dr.  Katherine  Morse, 

Johns  Hopkins 

University/APL 

8988  -  How  to  Save 

Time  and  Money  Using 

Lean  Maintenance 

Processes 

Mr.  Tim  Olson,  Lean 

Solutions  Institute,  Inc. 

8831  -  Human- Centered 

Design  in  Systems 

Engineering:  Human 

View  Methodology 

Dr.  Robert  Smillie, 

SPAWAR 
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9007  -  Technology 
Maturation  for  the 

Automated  Aerial 

Refueling  (AAR)  Project 

Ms.  Carol  Ventresca, 
SynGenics  Corporation 

9048  -  Revisions  to  the 
Acquisition  Modeling  & 

Simulation  Master  Plan 

Mr.  Stephen  Swenson, 
Systems  Engineering 
Directorate,  ODDR&E 

8908  -  Successful  First 
AESA  Deployment 
through  Application  of 
System  Engineering 

Mr.  Scott  Nichols, 

Raytheon  Company 

9097  -  Acquisition  ESOH 
Risk  Management-How  to 

Make  It  Work 

Mr.  Bob  Smith,  Booz 

Allen  Hamilton 

8980  -  Using  Model- 
driven  Engineering 
Techniques  for  Integrated 
Flight  Simulation 
Development 

Mr.  Douglas  Fiehler, 
Raytheon  Missile  Systems 

8950  -  Live,  Virtual, 

Constructive  Architecture 

Roadmap:  The  Quest  for 
Interoperability,  Standards, 

and  Reuse 

Dr.  Gary  Allen,  Joint 
Training  Integration  & 

Evaluation  Center 

9031  -  Supportability 

Lessons  Learned  with  Line 

Replaceable  Modules 

Ms.  Heity  Hsiung, 

Raytheon  Company 

8920  -  A  Methodology 
for  Assessing  Systems 
Engineering  Practices 

Ms.  Lauren  Levy,  Johns 
Hopkins  University/APL 

8875  -  Tomahawk  Weapon 
System  Development  and 
Integration 

Mr.  Gustavo  Rivera,  Naval 
Surface  Warfare  Center, 
Dahglren  Division 

8939  -  Understanding  the 

New  DoD  Instruction 

5000.61:  “DoD  Modeling 
&  Simulation  Verification, 

Validation  and  Accreditation 

(W&A)” 

Mr.  Michael  Truelove, 

Systems  Engineering 
Directorate,  ODDR&E 

9063  -  An  Integrated  RAM 
Approach  to  System  Design 
and  Support 

Mr.  Robert  Finlayson,  Johns 
Hopkins  University/APL 

8887  -  Achieving  a  Systems 
Engineering  Culture  in  a 
Science  and  Technology 
Laboratory  Environment 

Mr.  Robert  Rapson, 

Materials  and 

Manufacturing  Directorate, 

AFRL 
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8802  -  Open  Source  Technology 
for  Enterprise  Health 

Management 

8901  -  Review  Results  of  the 

NDIA/OSD  Software  Test 

Summit/Workshop 

9506  -  Software  Acquisition 
Management  Practical  Experience 
Mr.  James  Jones,  SSAI 

spa 

O 

CO 

^  U  s  -h  9 

43  Q  8  |  Q  o 

s  °  00  K 

Mr.  Edward  Beck,  CSC 

Mr.  Thomas  Wissink,  Lockheed 

Martin  IS&GS 

0000  -  Implementing  CMMI  on  a 
COTS  Modification  Effort 

Mr.  Dave  Castellano,  U.S.  Army 

TRACK 7 

Work  Force  Development 
Palm  I 

Dr.  Don  Gelosh,  Systems 
Engineering  Directorate, 
ODDR&E 

and  Mr.  Mike  Ucchino, 

U.S.  Air  Force  Center  for 
Systems  Engineering 

8956  -  Systems  Engineering 
Approach  to  Workforce 
Development 

Mr.  James  Miller,  U.S.  Air  Force 

9046  -  Developing  an 

Introductory  Systems  Engineering 
Practitioners  Course:  “Model- 
Based  Systems  Engineering 
(MBSE)  With  SysML” 

Mr.  Joseph  Wolfrom,  Johns 
Hopkins  University/ APL 

8878  -  Advanced  Simulation 

Course  for  Army  Simulation 
Management  Professionals 

Dr.  Gene  Paulo,  Naval 

Postgraduate  School 

TRACK 6 
Enterprise  Health 
Management 
Mission  III 

Mr.  Howard  Savage, 

Savage  Consulting  and 

Mr.  Chris  Reisig,  The 

Boeing  Company 

8815  -  Applying  Systems  Engi¬ 
neering  to  Operational  System 
Improvements 

Ms.  Ryanne  Gentry,  Acquisition 
Logistics  Engineering 

8842  -  Applications  in  Integrated 
Diagnostics 

Mr.  Jimmy  Simmons,  Georgia 

Tech  Research  Institute 

8884  -  Tactical  Wheeled  Vehicle 
Integrated  Diagnostics 

Mr  Lawrence  Osentoski,  DRIVE 
Developments,  Inc. 

TRACK  5 

Systems  Engineering 
Development 
Environment 

Mission  II 

Mr.  A1  Brown,  The 

Boeing  Company 

8967  _  Generating  Visual  and 
Interactive  Output  from  System 
Engineering  Tools 

Mr.  John  Schatz,  Systems  and 
Proposal  Engineering  Company 

9015  -  Challenges  and  Benefits  of 
applying  ISO  STEP 

Mr.  Stuart  Booth,  Systems 
Engineering  Directorate, 

ODDR&E 

9059  "  Smallsat  Conceptual 

Design  Trade  and  Cost  Modeling 

Tool 

Dr.  Deganit  Armon,  Advatech 
Pacific,  Inc 

TRACK 4 
Practical  Systems 
Engineering 
Mission  I 

Mr.  Dana  Peterson, 

DRS  Technologies,  Inc. 

8976  -  A  Systems  Engineering 
Model  for  Roadmap  Alignment 

Mr.  Si  Dok,  U.  S.  Army 

TARDEC 

9080  -  Rapid  Systems  Engineering 

of  the  MRAP  Gunner  Restraint 

System  Saves  Lives 

Ms.  Michelle  Bowen,  JPO  MRAP 
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AWARD  INFORMATION 

2009  LT  GEN  THOMAS  R.  FERGUSON,  JR. 

SYSTEMS  ENGINEERING  EXCELLENCE  AWARD 

The  National  Defense  Industrial  Associations  Systems  Engineering  Excellence  Awards  were  established  in 
2003  to  honor  the  memory  of  Lt  Gen  Thomas  R.  Ferguson,  Jr.,  USAF,  whose  leadership  embodied  the 
highest  ideals  in  Defense  Systems  development  and  deployment. 

The  awards  are  given  to  an  individual  and  to  a  group  demonstrating  outstanding  achievement  in  the 
practical  application  of  Systems  Engineering  principles,  promotion  of  robust  systems  engineering  principles 
throughout  the  organization,  or  effective  systems  engineering  process  development  during  the  previous 
year.  Their  systems  engineering  contributions  should  have  demonstrably  helped  achieve  significant  cost 
savings  due  to  new  or  enhanced  processes  procedures  and/or  concepts,  increased  mission  capabilities,  or 
substantially  increased  performance.  The  2009  awardees  are: 


►  Systems  Engineering  Individual  Leadership  Award:  Mr.  Brian  Wells 

►  Systems  Engineering  Group  Award:  Center  for  Advanced  Life  Cycle  Engineering 


PAST  AWARD  WINNERS: 


2003: 

►  Systems 
2004: 

►  Systems 
2005: 

►  Systems 
2006: 

►  Systems 

►  Systems 

►  Systems 
2007: 

►  Systems 

►  Systems 
2008: 

►  Systems 

►  Systems 


Engineering  Individual  Leadership  Award:  Mr.  Robert  Rassa 

Engineering  Individual  Leadership  Award:  Honorable  Mike  Wynne 

Engineering  Individual  Leadership  Award:  Mr.  Mark  Schaeffer 

Engineering  Individual  Leadership  Award:  Mr.  Kelly  Miller 

Engineering  Individual  Practitioner  Award:  Mr.  David  Strimling 

Engineering  Group  Award:  NUWC  Division  Newport  Critical  Transducer  Program  Staff 

Engineering  Individual  Leadership  Award:  Mr.  Robert  Skalamera 
Engineering  Group  Award:  Submarine  Warfare  Federated  Tactical  System  Team 

Engineering  Individual  Leadership  Award:  Honorable  James  Finley 

Engineering  Group  Award:  Tactical  Direction  Agent  Team  for  LCS  Mission  Package  Project 
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AWARD  INFORMATION 

DEPARTMENT  OF  DEFENSE  AND  THE  NATIONAL  DEFENSE  INDUSTRIAL  ASSOCIATION 
2008  TOP  5  DEPARTMENT  OF  DEFENSE  PROGRAM  AWARDS 

The  Department  of  Defense  Executive  Agent  for  Systems  Engineering  and  the  Systems  Engineering  Division  of 
the  National  Defense  Industrial  Association  are  pleased  to  announce  the  selections  of  the  2008  Top  5  Department 
of  Defense  Program  Awards.  The  2008  Program  awardees  are: 

►  Wideband  Global  SAT COM:  U.S.  Air  Force  PM;  Boeing  Company  Space  &  Intelligence  Systems  Group 

►  Joint  Light  Tactical  Vehicle:  U.S.  Army/USMC  PMs;  BAE  Systems  Land  &  Armaments;  General  Tactical 
Vehicles;  Lockheed  Martin  Systems  Integration 

►  STRYKER  Modernization:  U.S.  Army  PM;  General  Dynamics  Land  Systems 

►  Broad  Area  Maritime  Surveillance  Unmanned  Aircraft:  U.S.  Navy  PM;  Northrop  Grumman  Corporation 

►  Aviation  Maintenance  Training  Continuum  System:  U.S.  Navy  PM;  Raytheon  Company;  Paladin  Data 
Systems  Corporation 

The  Awards  are  presented  to  both  the  DoD  project  office  and  the  industry  prime  contractor  in  recognition  of 
total  program  performance  in  a  DoD/industry  team  effort. 

PAST  AWARD  WINNERS: 

2005  Top  5  Department  of  Defense  Programs: 

►  Centaur 

►  Integrated  Exploitation  Capability 

►  P-8A  Multi  Mission  Maritime  Aircraft 

►  Mission  INtegration  &  Development 

►  Tomahawk  Weapons  System  Program  PMA-280 

2006  Top  5  Department  of  Defense  Programs: 

►  Advanced  Extremely  High  Frequency  Mission  Control  System 

►  Advanced  Field  Artillery  Tactical  Data  System 

►  DDG  1000  MK57  Vertical  Landing  System 

►  Portable  Excalibur  ECS 

2007  Top  5  Department  of  Defense  Programs: 

►  Effects  Management  Tool 

►  MH-60  R/S  Link  16 

►  Mortar  Fire  Control  System  -  Dismounted 


SYSTEMS  ENGINEERING  CONFERENCE 
PROMOTIONAL  PARTNERSHIP 


THANK  YOU  TO  OUR  PROMOTIONAL  PARTNERS: 


PTC  provides  product  lifecycle  management  solutions  designed  to  meet  the 
requirements  of  the  global  aerospace  and  defense  industry.  These  solutions 
enable  digital  automation  of  product  development  and  program  management 
processes,  complete  visibility  and  control  over  program  information  for  secure, 
collaborative  product  development  as  well  as  dynamic  publishing  that  allows  you  to  produce  vital  service  information  directly 
in  the  standards-based  formats  -  either  in  print  or  on  the  Web.  PTC  is  an  industry  leader,  serving  the  product  development 
needs  of  the  top  20  A&D  companies.  Further  information  is  available  at  http://www.ptc.com/go/a-d. 


^University  of  Phoenix® 

*\T 


At  University  of  Phoenix,  we've  been  thinking  ahead  for  more  than  30 
years.  In  fact,  we  were  founded  in  1976  on  an  innovative  idea:  make 
higher  education  highly  accessible  for  working  students. 

Still  guided  by  this  idea,  University  of  Phoenix  has  helped  transform  the 
landscape  of  higher  education  in  widely  recognized  ways. 


Many  of  the  conveniences  that  21st-century  students  now  take  for  granted — evening  classes,  flexible  scheduling,  continuous 
enrollment,  a  student-centered  environment,  practitioner  faculty,  online  classes,  online  library,  ebooks,  computer  simulations — 
were  pioneered  or  made  acceptable  through  University  of  Phoenix's  efforts. 

Configuration  Management  Data  Management  Coursework 


This  program  exposes  students  to  the  most  important  principles  concerning  configuration  management  history,  configuration 
identification,  configuration  change  management,  and  data  management.  Courses  are  available  over  the  internet  through  our 
Online  Learning  System  (OLS)  or,  in  small  classes  at  select  classroom  locations  as  available. 


To  learn  more  contact  University  of  Phoenix  -  Center  for  Professional  Development  at  1  (800)  325-1509  or  via  email 
prodev@phoenix.edu. 


SYSTEMS  ENGINEERING  CONFERENCE 
PROMOTIONAL  PARTNERSHIP 


Lean  Solutions  Institute,  Inc.  (LSI)  specializes  in 

LEAN  SOLUTIONS  INSTITUTE,  Inc.  helping  organizations  to  rapidly  achieve  measur- 

LEAN  SOLUTIONS™  FOR  TOUR  ORGANIZATION  fb'e  results  by  using  benchmarking  and  Lean 

SolutionsTM  (e.g.,  best  practices  to  implement 
CMMI®  in  a  lean  way)  to  successfully  improve 
client  products  and  services.  LSI  helps  organiza¬ 
tions  to  measurably: 

•  Achieve  ROI  (e.g.,  7:1) 

•  Increase  productivity,  performance  and  quality 

•  Reduce  cycle  time/schedule 

•  Reduce  defects  (e.g.,  post-release  defects),  rework  and  costs  of  poor  quality 

•  Achieve  world-class  results  (e.g.,  70-90%  defect  removal  efficiency  or  defects  removed  before  test) 

Systems  engineering  and  software  engineering  have  become  more  and  more  complex  over  the  years.  With  this  growing 
complexity,  processes  and  procedures  have  become  larger  and  more  complex.  Based  on  surveys,  most  organizations  do  not 
like  their  processes  and  procedures  (e.g.,  including  CMMI®  Maturity  Level  3-5  organizations)  and  they  can  have  some  of  the 
following  lean  problems: 

•  Too  large  and  complex  (i.e.,  not  lean  or  agile) 

•  Have  non-value  added  activities 

•  Lack  of  visualization  (e.g.,  pictures,  diagrams,  tables,  charts,  etc.) 

•  Difficult  to  use  (e.g.,  poor  usability) 

•  Lack  of  “chunking”  which  is  a  best  practice  for  usability  (7  plus  or  minus  2  principle) 

•  Lack  of  innovation 

•  Lack  of  “good  metrics”,  not  the  right  metrics,  or  not  lean  metrics 

LSI  has  a  patent  pending  approach  for  defining  systems  engineering  and  software  engineering  processes  (e.g.,  CMMI®  com¬ 
pliant  processes)  in  a  lean  (e.g.,  short,  usable,  visual)  way.  Although  this  approach  can  be  simple,  it  also  scales  up  to  handle 
complex  processes  (e.g.,  NASA  processes).  LSI  uses  “good  diagrams”  (i.e.,  process  models)  for  putting  the  5  Ws  (who,  what, 
where,  when,  why)  on  one  page.  These  visual  one-page  diagrams  along  with  a  page  of  support  text  typically  replace  about  25- 
30  pages  of  text.  For  example,  lean  CMMI®  processes  are  typically  about  20-25%  of  the  size  of  a  typical  CMMI®  implemen¬ 
tation,  and  take  half  the  time  to  implement  (e.g.,  1  year).  In  several  CMMI®  success  stories  (independently  verified)  using  the 
LSI  approach,  organizations  estimate  that  processes  are  about  20%  of  the  size  of  sister  business  units  with  a  similar  CMMI® 
rated  processes,  and  have  achieved  CMMI  maturity  levels  half  the  time  (or  less). 

LSI  can  help  your  organization  achieve  measurable  results,  reduce  size  and  complexity,  and  improve  processes  and  metrics 
to  become  much  more  lean,  “value  added”,  visual,  and  usable.  LSI  also  uses  an  ISO/Baldrige  approach  to  implementing 
CMMI®.  LSI  only  does  improvement  and  uses  independent  Authorized  SEI  Lead  Appraisers  to  objectively  verify  LSI  Lean 
SolutionsTM  for  CMMI®. 


CMMI  is  a  registered  trademark  of  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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Requirements  and  Architecture 
Challenges1 


Requirements  and  Architecture  are  the  first  two  Opportunities  to  make 
Major  Engineering  Mistakes. 

Architecturally  Significant  Requirements  are  typically  poorly  engineered. 

Architecture  and  associated  Architecturally  Significant  Requirements 
Affect: 

•  Project  Organization  and  Staffing  (Conway’s  Law) 

•  Downstream  Design,  Implementation,  Integration,  Testing,  and  Deployment 
Decisions 

A  common  project-specific  Quality  Model  is  needed  to  drive  the 

•  Quality  Requirements,  which  drives  the 

•  Quality  of  the  System  Architecture,  which  drives  the 

•  Quality  of  the  System 
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Requirements  and  Architecture 
Challenges2 


Architecturally-Significant,  Quality-Related  Requirements  and 
their  associated  Architectural  Decisions  Drive  the  System  and 
Component: 

•  Ultimate  Quality 

•  Development  Schedule 

•  Development  Costs 

•  Sustainment  Costs 

•  Maintainability  and  Upgradeability 

•  Acceptance  and  Usage  by  Stakeholders 
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Requirements  and  Architecture 
Challenges3 


It  is  important  to  identify  (and  thereby  help  Manage)  Risks: 

•  Requirements  and  Architecture  Risks 

•  System  and  Project  Risks 

•  Business  Risks 

It  is  important  to  provide  Acquirer/Management: 

•  Visibility  into 

•  Oversight  over 

the  System  and  Component  Requirements  and  Architecture 

It  is  important  to  determine  Compliance : 

•  Requirements  and  Architecture  with  Contract  (Acquirer)  Requirements 

•  Architecture  with  System  and  Component  (Developer)  Requirements 
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What  is  Quality? 


Quality 

the  Degree  to  which  a  Work  Product  (e.g.,  System,  Subsystem, 
Requirements,  Architecture)  Exhibits  a  Desired  or  Required  Amount 
of  Useful  or  Needed  Characteristics  and  Attributes 

Not  just  lack  of  defects! 

Question: 

What  Types  of  Characteristics  and  Attributes  are  these? 

Answer: 

They  are  the  Characteristics  defined  by  the  Project  Quality  Model. 
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Quality  Mode^ 


Quality  of  a  Work  Product  is  defined  in  terms  of  a  Quality  Model: 

•  Quality  Characteristics 

(a.k.a.,  Quality  Factors,  the  ‘ilities’) 

(e.g.,  availability,  extensibility,  interoperability,  maintainability, 
performance,  portability,  reliability,  safety,  security,  and 
usability) 

•  Quality  Attributes 

(a.k.a.,  Quality  Subfactors) 

(e.g.,  the  quality  attributes  of  performance  are  jitter,  latency, 
response  time,  schedulability,  throughput) 

•  Quality  Measurement  Scales 

(e.g.,  milliseconds,  transactions  per  second) 
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Quality  Model2 
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Quality  Model  -  Quality  Characteristics 
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Quality  Model  - 
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Quality  Case  -  Definition 


Quality  Case 

a  Cohesive  Collection  of  Claims,  Arguments,  and  Evidence  that 
Makes  the  Developers’  Case  that  their  Work  Product(s)  have 
Sufficient  Quality 

Foundational  Concept  underlying  QUASAR 

A  Generalization  and  Specialization  of  Safety  Cases  from  the 
Safety  Community: 

More)  Can  Address  any  Quality  Characteristic  and/or  Quality  Attribute 
Less)  May  be  Restricted  to  only  Requirements  or  Architecture 
Useful  for: 

•  Assessing  Quality 

•  System  Certification  and  Accreditation  (e.g.,  safety  and  security) 
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Quality  Cases  -  Components1 


A  Quality  Case  consists  of  the  following  types  of  Components: 

1.  Claims 

Developers’  Claims  that  their  Work  Products  have  Sufficient  Quality, 
whereby  quality  is  defined  in  terms  of  the  qualify  characteristics  and 
quality  attributes  defined  in  the  official  project  quality  model 

2.  Arguments 

Clear,  Compelling,  and  Relevant  Developer  Arguments  Justifying  the 
Assessors’  Belief  in  the  Developers’  Claims 
(e.g.,  decisions,  inventions,  trade-offs,  analysis  and  simulation 
results,  assumptions,  and  associated  rationales) 

3.  Evidence 

Adequate  Credible  Evidence  Supporting  the  Developers’  Arguments 
(e.g.,  official  project  diagrams,  models,  requirements  specifications 
and  architecture  documents;  requirements  repositories;  analysis  and 
simulation  reports;  test  results;  and  demonstrations  witnessed  by  the 
assessors) 
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Quality  Cases  -  Components2 
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Specialized  QUASAR  Quality  Cases 


QUASAR  utilizes  the  following  specialized  types  of 
Quality  Cases: 

1 .  Requirements  Quality  Cases 

2.  Architectural  Quality  Cases 


QUASAR  Version  1  only  had  Architectural  Quality 
Cases. 

QUASAR  Versions  2  and  3  have  Both  Types  of  Quality 
Cases. 
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QUASAR  Quality  Case  Responsibilities 


Requirements  Engineers  and  Architects’  Responsibilities: 

•  Prepare  Quality  Cases 

•  Provide  Preparation  Materials  (including  Presentation  Materials 
and  Quality  Cases)  to  Assessors  Prior  to  Assessment  Meetings 

•  Present  Quality  Cases  (Make  their  Case  to  the  Assessors) 

•  Answer  Assessors’  Questions 

Assessor  Responsibilities: 

•  Prepare  for  Assessments 

•  Actively  Probe  Quality  Cases 

•  Develop  Consensus  regarding  Assessment  Results 

•  Determine  and  Report  Assessment  Results: 

—  Present  Outbriefs 
—  Publish  Reports 
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Quality  Case  Diagram  Notation 
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Architectural  Interoperability  Case  Diagram 
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Example  QUASAR  Scope  - 
Four  Assessments 
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What  is  a  System  Architecture?1 


System  Architecture 

the  Most  Important,  Pervasive,  Top-Level,  Strategic  Decisions, 
Inventions,  Engineering  Trade-Offs,  Assumptions,  and  associated 
Rationales  about  How  a  System’s  Architectural  Elements  will 
collaborate  to  meet  the  System’s  Derived  and  Allocated 
Requirements 
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What  is  a  System  Architecture?2 


System  Architecture  Includes: 

•  The  System’s  Numerous  Static  and  Dynamic,  Logical  and 
Physical  Structures 

(i.e.,  Essential  Architectural  Elements,  their  Relationships,  their 
Associated  Blackbox  Characteristics  and  Behavior,  and  how  they 
Collaborate  to  Support  the  System’s  Mission  and  Requirements) 

•  Architectural  Decisions,  Inventions,  and  Tradeoffs 

(e.g.,  Styles,  Patterns,  and  Mechanisms  used  to  ensure  that  the 
System  Achieves  its  Architecturally-Significant  Product  and  Process 
Requirements  (esp.  Quality  Requirements  or  ‘ilities’) 

•  Strategic  and  Pervasive  Design-Level  Decisions 

(e.g.,  using  a  Design  Paradigm  such  as  Object-Orientation  or 
Mandated  Widespread  use  of  common  Design  Patterns) 

•  Strategic  and  Pervasive  Implementation-Level  Decisions 

(e.g.,  using  a  Safe  Subset  of  C++) 
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Some  Example  Views  of  Models  of  Structures 


Services 


View 
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Architecture  vs.  Design 


Architecture 

Design 

Pervasive  (Multiple  Components) 

Local  (Single  Components) 

Strategic  Decisions  and  Inventions  j 

Tactical  Decisions  and  Inventions 

Higher-Levels  of  System 

Lower-Levels  of  System 

Huge  Impact  on  Quality,  Cost,  &  Schedule 

Small  Impact  on  Quality,  Cost,  &  Schedule 

Drives  Design  and  Integration  Testing 

Drives  Implementation  and  Unit  Testing 

Driven  by  Requirements  and  Higher-Level 
Architecture 

Driven  by  Requirements,  Architecture,  and 
Higher-Level  Design 

Mirrors  Top-Level  Development  Team 
Organization  (Conway’s  Law)  | 

No  Impact  on 

Top-Level  Development  Team  Organization 
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Architectural  Documentation  Current-State 


System  Architecture  Documents: 

•  Mostly  natural  language  Text  with  Visio-like  Diagrams  (Cartoons) 

•  Logical  (functional)  and  Physical  Architecture 

DOD  Architecture  Framework  (DODAF): 

•  All-Views,  Operational  Views,  Systems  Views,  and  Technical  Standards 
Views  for  allocating  Responsibilities  to  Systems  and  Supporting  System 
Interoperability 

Models  (both  static  and  dynamic;  logical  and  physical): 

•  Tailored  UML  becoming  de  facto  Industry  Standard 

•  SysML  starting  to  become  Popular 

Visio  Diagrams  as  Wall  Posters 

Whitepapers,  Reports,  and  other  Specialty-Engineering  Documents: 

•  Performance,  Fault  Tolerance,  Reliability,  Safety,  Security 
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Quality  Requirements  -  Components 
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Topics 
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QUASAR  Method 
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Definition 


QUalitv  Assessment  of  System  Architectures  and  their  Requirements 

a  Well-Documented  and  Proven  Method  based  on  the  use  of  Quality 
Cases  for  Independently  Assessing  the  Quality  of: 

•  Software-intensive  System  /  Subsystem  Architectures  and  the 

•  Architecturally  Significant  Requirements  that  Drive  Them 
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QUASAR  Philosophy., 


Informal  Peer  Reviews  are  Inadequate: 

•  Too  Informal 

•  Lack  of  Independent  Expert  Input 

•  Requirements  and  Architecture  are  too  Important 

Quality  Requirements: 

•  Most  important  Architecturally-Significant  Requirements 

•  Largely  Drive  the  System  Architecture 

•  Criteria  against  which  the  System  Architecture  is  Assessed 
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QUASAR  Philosophy2 


Requirements  Engineers  (REs)  should  Make  Case  to  Assessors: 

•  REs  should  know  Stakeholder  Needs  and  Goals 

•  REs  should  know  What  they  Did  and  Why 
(Architecturally-Significant  Requirements,  Rationales,  &  Assumptions) 

•  REs  should  Know  Where  they  Documented  their  Requirements  Work 
Products 

Architects  should  Make  Case  to  Assessors: 


•  Architects  should  know  Architecturally-Significant  Requirements 

•  Architects  should  know  What  they  Did  and  Why 

(Decisions,  Inventions,  Trade-Offs,  Assumptions,  and  Rationales) 

•  Architects  should  know  Where  they  Documented  their  Architectural 
Work  Products 
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QUASAR  Philosophy3 


Assessors  should  Actively  Probe  Quality  Cases: 

•  Claims  Correct  and  Complete? 

Do  the  Claims  include  all  relevant  Quality  Characteristics,  Quality 
Attributes,  Quality  Goals,  and  Quality  Requirements? 

•  Arguments  Correct,  Complete,  Clear,  and  Compelling? 

Do  the  Arguments  include  all  relevant  Quality  Characteristics,  Quality 
Attributes,  Quality  Goals,  Quality  Requirements,  Decisions, 
Inventions,  Trade-offs,  Assumptions,  and  Rationales? 

•  Arguments  Sufficient? 

Are  the  Arguments  Sufficient  to  Justify  the  Claims? 

•  Evidence  Sufficient? 

Is  the  Evidence  Sufficient  to  Support  the  Arguments? 

•  Current  Point  in  the  Schedule? 

Are  the  Claims,  Arguments,  and  Evidence  appropriate  for  the 
Current  Point  in  the  Schedule? 
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QUASAR  Method  -  Three  Phases 


1 .  Quality  Assessment  Initiation  (QAI) 

2.  Requirements  Quality  Assessment  (RQA) 


3.  Architecture  Quality  Assessment  (AQA) 


Quality 
Assessment 
Initiation 


repeat  for  system  and  each  subsystem  being  assessed 


Requirements 
Quality 
Assessment 


Architecture 

Quality 

Assessment 
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QUASAR  Phases  and  Tasks 


Phase  1)  Quality 
Assessment  Initiation  (QAI) 

Prep. 

QAI 

Meeting 

Follow- 

Through 

Time  (not  to  scale) 


System  Assessments 


Phase  2)  Requirements  Quality 
Assessment  (RQA) 

Prep. 

RQA 

Meeting 

Follow- 

Through 

Phase  3)  Architecture  Quality 
Assessment  (AQA) 

Prep. 

AQA 

Meeting 

Follow- 

Through 

Subsystem  1  Assessments 


Phase  2)  Requirements  Quality 
Assessment  (RQA) 

Prep. 

RQA 

Meeting 

Follow- 

Through 

Phase  3)  Architecture  Quality 
Assessment  (AQA) 

Prep. 

AQA 

Meeting 

Follow- 

Through 

Subsystem  N  Assessments 


Phase  2)  Requirements  Quality 
Assessment  (RQA) 

Prep. 

RQA 

Meeting 

Follow- 

Through 

Phase  3)  Architecture  Quality 
Assessment  (AQA) 

Prep. 

AQA 

Meeting 

Follow- 

Through 

-  Software  Engineering  Institute 


Carnegie  Mellon 


QUASAR  Version  3.1  Overview 
Donald  Firesmith,  29  October  2009 

©  2009  Carnegie  Mellon  University 


32 


Quasar  Teams  and  their  Work  Products 


System 

Requirements 

Team 


Top-Level 

Architecture 

Team 
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Quality  Assessment  Initiation  (QAI) 


Quality 
Assessment 
Initiation 


repeat  for  system  and  each  subsystem  being  assessed 


Requirements 

Quality 

Assessment 


Architecture 

Quality 

Assessment 
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Phase  1)  QAI  -  Objectives 


Prepare  Teams  for  Requirements  and  Architecture  Assessments 

Develop  Consensus: 

•  Scope  of  Assessments 

•  Schedule  Assessments 

•  Tailor  the  Assessment  Method  and  associated  Training  Materials 
Produce  and  Publish  Meeting  Outbrief  and  Minutes 
Manage  Action  Items 

Capture  Lessons  Learned 

Tailor/Update  QUASAR  Method  and  Training  Materials 
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Phase  1)  QAI  -  Preparation  Task 


1 .  Management  Team  staffs  Assessment  Team 

2.  Process  and  Training  Teams  train  Assessment  Team 

3.  Assessment  Team  identifies: 

•  System  Requirements  Team 

•  System  Architecture  Team 

4.  Process  and  Training  Teams  train  System  Requirements 
and  Architecture  Teams 

5.  Assessment,  Requirements,  and  Architecture  Teams 
collaborate  to  Organize  QAI  Meeting 

(i.e.,  Attendees,  Time,  Location,  Agenda) 
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Phase  1)  QAI  -  Meeting  Task 


1 .  Assessment,  System  Requirements,  and  System  Architecture 
Teams  Collaborate  to  determine  Assessment  Scope: 

•  Subsystems/Architectural  Elements/Focus  Areas  to  Assess  (Number 
and  Identity) 

•  Quality  Characteristics  and  Quality  Attributes  underlying  Assessment 

•  Assessment  Resources  (e.g.,  Staffing,  Schedule,  and  Budget) 

2.  Teams  Collaborate  to  develop  Initial  Assessment  Schedule  with 
regard  to  System  schedule,  Subsystem  schedule,  and  associated 
milestones 

3.  Teams  Collaborate  to  tailor  QUASAR  Method 

4.  Assessment  Team  captures  Action  Items 
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Phase  1)  QAI  -  Follow-Through  Task 


1 .  Assessment  Team  develops  and  presents  Meeting  Outbrief 

2.  Assessment  Team  develops,  reviews,  and  distributes 
Meeting  Minutes 

3.  Assessment/Process/Training  Teams  tailor,  internally 
review,  and  distribute: 

•  QUASAR  Procedure,  Standards,  and  Templates 

•  QUASAR  Training  Materials 

4.  Teams  distribute  Assessment  Schedule 

5.  Teams  obtain  Needed  Resources 

6.  Assessment  Team  Manages  Action  Items 

7.  Assessment  Team  captures  Lessons  Learned 
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Phase  1)  QAI  -  Work  Product  Flow 
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Team 
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Phase  1)  QAI  -  Work  Products 
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Phase  1)  QAI  -  Lessons  Learned1 


Ensure  Appropriate  Team  Memberships  (e.g.,  Authority) 

Ensure  Adequate  Resources  (e.g.,  Staffing,  Budget,  and  Schedule) 
Obtain  Consensus  on: 

•  Assessment  Objectives  and  Scope 

•  Definitions  (e.g.,  of  Quality  Characteristics,  Attributes,  and  Cases) 

Provide  Early  Training: 

•  Method  Training 

(QUASAR,  Requirements  Engineering,  and  Architecting) 

•  System/Subsystem  Training 
(Requirements  and  Architecture) 
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Phase  1 )  QAI  -  Lessons  Learned2 


QUASAR  assessments  should  be  Organized  according  to  a 
Quality  Model  that  defines  Quality  Characteristics  (a.k.a.,  factors, 
“ilities’)  and  their  Quality  Attributes  such  as: 

•  Availability 

•  Interoperability 

•  Performance 

—  Jitter,  Response  Time,  Schedulability,  and  Throughput 

•  Portability 

•  Reliability 

•  Safety 

•  Security 

•  Usability 
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Requirements  Quality  Assessment 
(RQA) 


Quality 
Assessment 
Initiation 


repeat  for  system  and  each  subsystem  being  assessed 


Requirements 

Quality 

Assessment 


Architecture 

Quality 

Assessment 
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Phase  2)  ARA  -  Objectives1 


Use  Requirements  Quality  Cases  to: 

•  Independently  assess  Quality  and  Maturity  of  the  Architecturally 
Significant  Requirements: 

—  Drive  the  Architecture 

—  Form  Foundation  for  Architecture  Quality  Assessment 

•  Help  Requirements  Engineers  identify  Requirements  Defects  and 
Weaknesses  so  that: 

—  Defects  and  Weaknesses  can  be  Corrected 
—  The  Architecture  (and  System)  can  be  Improved 


-  Software  Engineering  Institute 
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Phase  2)  RQA  -  Objectives2 


Use  Requirements  Quality  Cases  to: 

•  Identify  Requirements  Risks  so  that  they  can  be  Managed 

•  Provide  Visibility  into  the  Status  and  Maturity  of  the  Requirements 

•  Increase  the  Probability  of  Project  Success 

Ensure  Architecture  Team  will  be  Prepared  to  Support  the  coming 
Architecture  Quality  Assessment. 

Capture  Lessons  Learned. 

Update  QUASAR  Method  and  associated  Training  Materials. 
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Phase  2)  RQA  -  Preparation  Task 


Process/Training  Team  trains  the  Requirements  and  Architecture 
Teams  significantly  prior  to  the  RQA  Meeting. 

Requirements  and  Architecture  Teams  provide  Preparatory  Materials  to 
the  Quality  Assessment  Team  significantly  prior  to  the  RQA  Meeting: 

•  Summary  Presentation  Materials 

•  Requirements  Quality  Cases 

(including  electronic  access  to  evidentiary  materials) 

•  Example  of  Planned  Architectural  Quality  Case 

Quality  Assessment  Team: 

•  Reads  Preparatory  Materials 

•  Generates  RFIs  and  RFAs 
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Phase  2)  RQA  -  Meeting  Task 


1.  Requirements  Team  presents: 

•  System  Overview 

•  Requirements  Overview 

•  Requirements  Quality  Cases 

2.  Quality  Assessment  Team  assesses  Quality  and  Maturity  of 
Requirements: 

•  Completeness  of  Quality  Cases 

•  Quality  of  Quality  Cases 

3.  Architecture  Team  presents  Example  Architectural  Quality 
Case 

4.  Quality  Assessment  Team  recommends  Improvements 

5.  Quality  Assessment  Team  manages  Action  Items 
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Phase  2)  RQA  -  Follow-Through  Task 


Quality  Assessment  Team: 

1.  Develops  Consensus  Regarding  Requirements  Quality 

2.  Produces,  Reviews,  and  Presents  Meeting  Outbrief 

3.  Produces,  Reviews,  and  Publishes  RQA  Report 

4.  Updates  and  publishes  the  System  Quality  Assessment  Summary 
Matrix 

5.  Captures  Lessons  Learned 

6.  Manages  Action  Items 
Requirements  Team: 

Addresses  Risks  Raised  in  RQA  Report 

Process  Team: 

Updates  Assessment  Method  (e.g.,  Standards  and  Procedures) 

Training  Team: 

Updates  Training  Materials  (if  appropriate) 
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Phase  2)  RQA  -  Work  Product  Workflow 


Requirements 
Team 
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Architecture 
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QUASAR 

Training  Materials 
QUASAR 

Stds  &  Procedures 

RQA  Checklist  and 
Report  Template 

RQA  Preparatory 
Materials 


Training 

Team 


RQA  Presentation 
Materials 
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Requirements 
Quality  Cases  and 
Draft  Representative 
Architectural 
Quality  Case 

Questions/Answers 


Recommendations 


RQA  Outbrief 


RQA  Report 


Action  Item  List 
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Phase  2)  RQA  -  Work  Products 
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System  Quality  Assessment 
Summary  Matilx 
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Architecture  Quality  Assessment  (AQA) 


Quality 
Assessment 
Initiation 


repeat  for  system  and  each  subsystem  being  assessed 
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Phase  3)  AQA  -  Objectives 


Use  Architectural  Quality  Cases  to: 

•  Independently  assess  Architecture  Quality  in  terms  of  its  Support  for 
its  Derived  and  Allocated  Architecturally  Significant  Requirements 

•  Help  Architects  identify  Architectural  Defects  and  Weaknesses  so 
that: 

—  Defects  and  Weaknesses  can  be  Corrected 
—  The  Architecture  (and  System)  can  be  Improved 

•  Identify  Architectural  Risks  so  that  they  can  be  Managed 

•  Provide  Visibility  into  the  Status  and  Maturity  of  the  Architecture 

•  Increase  the  Probability  of  Project  Success 
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Phase  3)  AQA  -  Preparation  Task 


Architecture  and  Quality  Assessment  Teams  organize  the  AQA 
Assessment  Meeting. 

Training  Team  provides  (at  appropriate  time): 

•  QUASAR  Training  (if  not  provided  prior  to  RQA  assessment) 

•  AQA  Assessment  Checklist  and  Report  Template 

Architecture  Team  makes  available  (min.  2  weeks  before  meeting): 

•  Any  Updated  Quality  Requirements 

•  Architecture  Overview 

•  Quality  Case  Diagrams 

•  Architecture  Quality  Cases  (Claims,  Arguments,  and  Evidence) 

Quality  Assessment  Team: 

•  Reads  Preparatory  Materials 

•  Generates  RFIs  and  RFAs 
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Phase  3)  AQA  -  Meeting  Task 


Architecture  Team: 

1 .  Introduces  the  Architecture 

(e.g.,  Context  and  Major  Functions) 

2.  Briefly  reviews  the  Architecturally  Significant  Requirements 

3.  Briefly  summarizes  the  Architecture 

(e.g.,  Most  Important  Architectural  Components,  Relationships, 
Decisions,  Inventions,  Trade-Offs,  Assumptions,  and  Rationales) 

4.  Individually  Presents  Architectural  Quality  Cases 
(Quality  Case  Diagram,  Claims,  Arguments,  and  Evidence) 

Quality  Assessment  Team: 

1 .  Probes  Architecture  (Architectural  Quality  Case  by  Quality  Case) 

2.  Manages  Action  Items 
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Phase  3)  AQA  -  Follow-Through  Task 


Quality  Assessment  Team: 

1 .  Develops  Consensus  regarding  Architecture  Quality 

2.  Produces,  reviews,  and  presents  Meeting  Outbrief 

3.  Produces,  reviews,  and  publishes  AQA  Report 

4.  Updates  and  republishes  System  Quality  Assessment  Summary 
Matrix 

5.  Captures  Lessons  Learned 

6.  Manages  Action  Items 
Architecture  Team: 

Addresses  Architectural  Defects,  Weaknesses,  and  Risks  Raised  in 
AQA  Report 
Process  Team: 

Updates  Assessment  Method  (if  appropriate) 

Training  Team: 

Updates  Training  Materials  (if  appropriate) 
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Phase  3)  AQA  -  Work  Product  Workflow 
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Phase  3)  AQA  -  Primary  Work  Products 
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Topics 


Requirements  and  Architecture  Challenges 
Underlying  Concepts 
QUASAR  Method 

Reasons  to  use  QUASAR 
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QUASAR  Benefits1 


QUASAR  ensures  Specification  of  Architecturally-Significant 
Requirements. 

QUASAR  provides  Acquirer  Visibility  into  (and  supports  oversight 
of)  the  Quality  of  the  Requirements  and  Architecture 

QUASAR  supports  Certification  and  Accreditation 

QUASAR  emphasizes  using  a  common  project-specific  Quality 
Model: 

•  Which  drives  the  Quality  Requirements 

•  Which  drives  the  Quality  of  the  System  Architecture 

•  Which  drives  the  Quality  of  the  System 


—  Software  Engineering  Institute 
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QUASAR  Benefits2 


QUASAR  Supports  Process  Improvement: 

•  Solves  Major  Requirements  and  Architecture  Problems 

QUASAR  Provides  needed  Flexibility: 

•  Any  Effective  Requirements  Engineering  and  Architecting  Methods 

•  Uses  Existing  Requirements  and  Architecture  Work  Products 
(i.e.,  almost  no  new  work  products  required) 

•  Any  Subsystems  based  in  Need  and  Risk 

(i.e.,  fits  any  system  size,  budget,  schedule,  and  tier) 

•  Any  Quality  Characteristics  and  Quality  Attributes 

QUASAR  Helps: 

•  Requirements  Engineers  Succeed 

•  Architects  Succeed 

•  Program  Succeed 
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How  the  SEI  Can  Help  You 


QUASAR  is  Ready  for  Use  Now. 

QUASAR  Handbook  and  Training  Materials  can  be  downloaded 
from  SEI  Website. 

The  SEI  Acquisition  Support  Program  (ASP)  offers  QUASAR  as  a 
Service: 

•  Consulting  and  Training 

•  Facilitation  of  QUASAR  Assessments 

•  Recommended  RFP  and  Contract  Language 
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Questions? 


For  more  information: 

Donald  Firesmith 
Acquisition  Support  Program 
Software  Engineering  Institute 
dgf@sei.cmu.edu 


The  Method  Framework  for  Engineering  System  Architectures 
(MFESA),  Donald  Firesmith  et  al.,  Auerbach  Publishing,  November 
2008 

Quasar  Tutorial  (1  day)  : 

http://www.sei.cmu.edu/library/abstracts/presentations/quasartutorial2 

008. cfm 
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r  Software  Engineering  Institute 
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Models  &  Simulations 
Development  Best 
Practices 


NDIA  Systems  Engineering  Conference 

29  October  2009 


•bnon  Vick  - -  7 - - 7 

•Nathaniel  Horner  APPLIED  PHYSICS  LABORATORY 


Presentation  Outline 


■  Background 

■  Study  Objectives  and  Major  Technical  Activities 

■  Survey  and  Literature  Search 

"  Best  Practice  Template  and  Example 

■  Simulation  Interoperability  Standards  Organization  (SISO)  Study  Group 

■  Systems  Engineering  Framework 
"  Literature  Search  Results 
■  Current  Status 

■  Integration  Plan  and  Evaluation  Criteria 

■  Best  Practices  Review  Status 
-  Planned  Next  Steps 

■  Discussion 
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dPL 


Background 


■  Although  the  importance  and  use  of  modeling  and  simulation  tools 
(models,  simulations,  and  utilities)  is  expanding  across  the  DoD, 
relatively  few  persons  have  a  good  grasp  of  the  process  and 
principles  that  should  be  followed  when  developing  such  tools. 

■  The  DoD  has  identified  the  Federation  Development  and 
Execution  Process  (FEDEP  -  IEEE  1516.3)  as  a  recommended 
practice  for  distributed  simulation  federations  using  the  HLA,  but 
no  equivalent  best  practice  exists  for  the  development  of 
individual  modeling  and  simulation  tools. 

■  Whether  conducting  such  a  development  or  overseeing  a 
contractor’s  efforts  to  do  so,  DoD  acquisition  professionals  need  to 
understand  best  practices  for  developing  modeling  and  simulation 
tools. 
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Study  Objectives  and 
Major  Technical  Activities 

■  Study  Objectives 

■  Identify  effective  practices  for  the  efficient  development  and 
evolution  of  credible  models  and  simulations 

■  Major  Technical  Activities 

■  Conduct  a  literature  search  and  survey  of  M&S  tool  developers  to 
identify  sound  practices  for  M&S  development 

■  Develop  an  overarching  systems  engineering  framework  for 
describing  the  activities  and  tasks  necessary  for  effective  M&S 
development 

■  Develop  a  plan  for  populating  the  SE  framework  with  the 
appropriate  process  elements  (activities  and  tasks),  and  for 
capturing  best  practices  specific  to  chosen  domain  areas 

■  Review  the  draft  framework  with  organizations  and  individuals 
that  can  help  ensure  its  correctness  and  appropriateness 

■  Refine  the  core  process  document  descriptions  per  the  above 
reviews 
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Literature  Search 


■  Assembled  bibliography  of  (mostly)  journal  and  book  sources 

■  Searched  NDIA,  Simulation  Interoperability  Workshop  (SIW)  and 
Interservice/Industry  Training,  Simulation  &  Education  Conference 
(l/ITSEC)  papers  from  the  last  5  years 

■  Literature  search  and  survey  together  resulted  in  approximately  116 
practices  for  consideration 


dPL 


Initial  Community  Survey 


1.  Does  your  organization  develop  models/simulations,  supporting 
environments  for  developing  models/simulations,  or  both? 

2.  Are  your  organization’s  practices  based  on  industry  standards  or 
internally  developed?  [Industry  standards  -  skip  to  Question  4] 

3.  Is  your  organization  willing  to  provide  a  detailed  description  of 
these  practices  to  the  JHU/APL  Study  Team,  assuming  any 
intellectual  property  is  properly  protected  by  a  non-disclosure 
agreement?  [Internally-developed  practices  stop  here] 

4.  Please  name  and  provide  appropriate  references  for  the  industry 
standards  upon  which  your  practices  are  based. 

5.  Please  describe  your  tailoring  of  the  industry  standards  for 
application  within  the  M&S  domain.  If  you  would  prefer  to 
discuss  this  with  the  study  team  under  a  non-disclosure 
agreement  (NDA)  to  protect  your  intellectual  property,  please  so 
indicate. 

- dPL 


Initial  Survey  Results 


■  47  respondents 

■  4  have  proprietary  practices  they  won’t  discuss  without  NDA 

■  Respondents  were  almost  evenly  split  between  using  industry 
standards  and  internally  developed  practices 

■  Most  respondents  develop  both  models/simulations  and  supporting 
environments 

■  There  was  some  confusion  on  the  question  about  industry 
standards  used  because  several  responded  with  HLA  and 
Distributed  Interactive  Simulation  (DIS) 

■  This  confusion  will  be  cleared  up  in  the  follow-on  conversations 

■  Fewer  than  half  of  respondents  answered  this  question  at  all 

■  CMMI  -  7;  ISO  9000/9001  -  5  (8?);  INCOSE  -  1;  EIA-632  - 1 
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Best  Practice  Template  with  Example 


Short  Descriptive  Title 


Consistent  intermediate  conceptual  model 


SE  Framework  Category 


PQC:  Name,  Email  Address,  Phone  #;  literature*  for  literature 


Requirements  engineering,  system 
design,  technical  overlays _ 


tu  re. 


Description 


A  well-conceived,  consistent  intermediate  [conceptual]  model  eliminates  many  problems  by 
providing  a  representation  of  the  usable  by  all  participants  (customer,  domain  expert, 

developer,  and  user).  Knowledge  objects  enable  the  certification  of  information  pedigree;  can  track 
changes  in  information;  provide  corrective  updates  when  necessary 


Radons  le  (Why  the  practice  Is  usefu  l/needed  0 


A  major  challenge  [to  developing  M&S  to  support  SE]  Is  creating  computationally  amenable 
descriptions  of  the  infinitely  rich  world  with  which  the  software  development  team  can  work.  There 
[is]  ajteaaqg&t  between  the  knowledge  management  and  SE  processes. _ 


Source  Reference:  If  derived  from  an  industry  standard,  provide  document  name  and  version,  and  section  number(s) 


Proceedings  of  the  9th  N DIA  Systems  Engineering  Conference,  "Training  for  Models:  The  Role  of 
Knowledge  Management  in  Applying  Modeling  and  Simulation  (M&S)  to  Systems  Engineering,"  David 
R.  Pratt  and  Robert  W.  Fra  nee  sc  bin! 


The  paper  was  about  a  proprietary  process,  but  the  use  of  an  intermediate  conceptual  model  is 
broadly  applicable. _ 


If  this  practice  Is  derived  from  another  source,  complete  the  sections  below. 


Description  of  T< 


SISO  Study  Group 


■  Formed  to  provide  input  and  feedback  to  study 

■  Potential  source  of  additional  information 

■  Tasks  and  deliverables  are  limited  to  review  and 
recommendations 

■  Is  a  necessary  first  step  in  the  SISO  process  if  we  want  the  results 
of  the  study  to  form  the  basis  of  a  SISO  standard 

■  Kickoff  meeting  at  the  Spring  SIW 

■  March  25,  2009 

■  San  Diego,  CA 


dPL 


Systems  Engineering  Framework 
Literature  Search  Results 


■  International  Council  on  Systems  Engineering  (INCOSE)  Handbook  (v3.1) 

■  Electronic  Industries  Alliance  (EIA)  Processes  for  Engineering  a  System 
(EIA-632) 

■  Institute  for  Electrical  and  Electronics  Engineers  (IEEE)  Standard  for 
Application  and  Management  of  the  Systems  Engineering  Process  (IEEE- 
1220) 

■  International  Organization  for  Standardization  (ISO)/lnternational 
Electrotechnical  Commission  (IEC)  Systems  engineering  -  System  life  cycle 
processes  (ISO/IEC-15288) 

■  Military  Standard  -  System  Engineering  Management  (MIL-STD-499C) 

■  IEEE  Federation  Development  and  Execution  Process  (FEDEP)  (IEEE 
1516.3-2003)/Distributed  Simulation  Engineering  and  Execution  Process 
(DSEEP)  (IEEE  P1730) 

■  Capability  Maturity  Model  Integration  (CMMI) 
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SE  Framework  Outline 


■  Phase  1 :  Requirements 
Development 

■  Activity  1 :  Develop  Stakeholder 
Requirements 

■  Activity  2:  Develop  and  Analyze  System 
Requirements 

■  Activity  3:  Validate  Requirements 

■  Phase  2:  Conceptual  Analysis 

■  Activity  1 :  Develop  Conceptual  Model 

■  Activity  2:  Validate  Conceptual  Model 

■  Phase  3:  Product  Design 

■  Activity  1 :  Perform  Functional  Analysis 

■  Activity  2:  Synthesize  Design 

■  Activity  3:  Verify  Design 


■  Phase  4:  Product  Development 

■  Activity  1 :  Establish  Software 
Development  Environment 

■  Activity  2:  Implement  Product  Design 

■  Phase  5:  Product  Testing 

■  Activity  1 :  Perform  Product  Verification 

■  Activity  2:  Perform  Product  Validation 

■  Project  Management  Practices 

■  Project  Planning 

■  Project  Control/Resource  Management 

■  R/'s/c  Management 

■  Quality  Management 

■  Configuration  Management 


API 


Integrating  Best  Practices  into  the  SE 
Framework 


1.  While  identifying  and  documenting  sound  practices,  the  study 
team  is  tagging  them  according  to  our  SE  framework  categories 
and  activities 

2.  The  team  has  developed  a  set  of  evaluation  criteria  (next  3  slides) 
for  selecting  best  practices  from  the  sound  practices 

3.  Once  the  best  practices  are  identified,  the  study  team  will  review 
the  practices  in  each  category,  shifting  them  to  other  categories  as 
necessary,  and  resolve  any  conflicts/overlaps  between  closely 
related  best  practices,  probably  merging  conflicts/overlaps  into  a 
single  practice 

4.  The  final  set  of  best  practices  will  be  assigned  by  consensus  of  the 
study  team  into  the  individual  activities  of  each  SE  category 

■  And,  of  course,  the  contributors  and  community  will  review  this 
assignment 


dPL 


Criteria  (1  of  3) 

■  Specificity  -  Does  the  practice  have  demonstrated  effectiveness 
within  specific  M&S  domains? 

■  Comparability  -  Has  the  practice  been  compared  positively  to 
other  practices  in  controlled  studies  (or  could  it  be)? 

■  Degree  of  Independence  -  Is  the  practice  platform  or 
implementation  independent? 

■  Efficacy  -  Does  the  practice  support  effective  use  of  resources 
including  intellectual  capital? 

■  Customization  -  Does  the  practice  allow  customization  and 
tailoring  to  an  organization  or  domain’s  needs? 

■  Coherence  -  Does  the  practice  align  with  other  adopted  best 
practices? 

■  Robustness  -  Does  the  practice  usually  result  in  a  better 
outcome? 


dPL 


Criteria  (2  of  3) 

■  Cohesion  -  Does  the  practice  describe  a  single  idea,  concept  or 
construct  and  not  multiple  ones  grouped  into  a  single  practice? 

■  Coupling  -  Is  the  practice’s  adoption  independent  of  other 
practices,  i.e.  does  the  adoption  of  this  practice  necessitate  the 
adoption  of  another? 

■  Sustainability  -  Is  it  cost  effective  to  sustain  the  practice  after 
adoption? 

■  Usability  -  Can  the  practice  be  used,  learned  and  employed  in 
practice? 

■  Scalability  -  Is  the  practice  scalable  to  projects  of  different  sizes? 

■  Agility  -  Can  the  practice  adapt  to  changing  conditions,  e.g. 
organization  changes,  contextual  changes,  etc.)  readily? 

■  Generality  -  Is  the  practice  expressed  as  generally  as  possible? 

■  Legal  aspects  -  Is  adoption  of  the  practice  free  of  difficult 
legal/proprietary  aspects? 
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Criteria  (3  of  3) 

■  Consensus  -  Is  there  widespread  community  acceptance  of  the 
practice? 

■  Cost  Elasticity  -  Do  the  benefits  of  the  results  outweigh  the  cost 
of  adoption  of  the  practice? 

■  Repeatability  -  Does  the  practice  repeatedly  give  desired 
results? 

■  Durability  -  Does  the  practice  remain  effective  over  time? 

■  Applicability  -  Is  the  technology  related  to  the  practice  widely 
applicable  and  not  just  to  a  subset  of  problems  or  domains? 
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Best  Practices  Review  Status 


■  Started  with  116 

■  Removed  those  that  restated  concepts  already  in  the  SE  Framework 

■  Approximately  10 

■  Team  members  individually: 

■  Assessed  practices  against  evaluation  criteria 

■  Assigned  practices  to  phases  and  activities  in  the  SE  Framework 

■  Assessed  whether  the  practices  were  M&S  specific 

■  Team  is  working  through  practices  in  batches,  debating  our 
positions  and  reaching  consensus 

■  Approximately  half  complete  and  making  good  progress 

■  Identified  the  need  to  clean  up  several  practices 

■  Transcription  errors 

■  Overlaps  between  practices 

■  Separating  rationale  from  practice 

- dPL 


Planned  Next  Steps 

■  Complete  SE  framework 

■  Complete  review  and  clean-up  of  practices 

■  Integrate  practices  into  framework 

■  Get  feedback  from  stakeholders  and  contributors  on  framework  and  best 
practices 


dPL 


Questions? 


dPL 


18 


Achieving  Acquisition  Excellence 
via  Improving  the  Systems- 
Engineering  Workforce 


Dr.  Kenneth  E.  Nidiffer 
Software  Engineering  Institute 
Carnegie  Mellon  University 

12th Annual  ndia  Pittsburgh,  PA  15213 

Systems  Engineering  Conference 
“Achieving  Acquisition  Excellence  via 
Effective  Systems  Engineering” 

San  Diego,  ca  October  29,  2009 

26-29  October 


Software  Engineering  Institute 


Carnegie  Mellon 


©2009  Carnegie  Mellon  University 


Overview 


•  Is  your  organization  working  towards  achieving 
acquisition  excellence? 

-  The  application  of  systems-engineering  to  improve  the 
workforce  may  be  part  of  the  answer! 

•  What  are  the  rate-limiting  variables/drivers  that 
limit  success? 

•  How  can  the  CMMI®-  ACQ  model  be  used? 


Achieving  Acquisition  Excellence  via  Effective  Application  of  CMMI® -ACQ 
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Procurement  Budget 

vs.  DoD  Acquisition  Workforce 
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Increasing  #  of  Procurements  &  Complex  Systems  Coupled  With 
Huge  Decrease  In  Acquisition  Workforce 
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Recapture  Acquisition  Excellence: 

Revitalize  The  Acquisition  Workforce 

Problem 

•  Acquisition  capability  has  slowly  atrophied 

•  Organic  Workforce  reductions  -  23%  since  1999 

>  Force  shaping,  reduced  training,  retirements  of  critical  cost 
estimators,  price  analysts,  experienced  system  engineers, 
contracting  officers 

Initiatives 

•  Recapitalize  the  Acquisition  Corps/Training 

•  OSD  Funding  Increased  Numbers  and  Training  of  Organic 
Acquisition  Personnel 

I _ It  Is  All  About  the  Acquisition  Workforce 
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Software  Engineering  Institute  Carnegie  Mellon 


Project  Purpose 

Use  a  systems  engineering  approach  to  assess  acquisition 
training  and  organizational  training  processes  for 
improving  acquisition  excellence 


Experience 
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Summary  of  Systems  Engineering  Drivers 

External  Forces 

•  Increasing  size  of  untrained  defense  acquisition  workforce 

•  Retiring  of  experienced  and  capable  workforce 

Technological 

•  Accelerating  technological  changes  makes  systems  specific  acquisition 
training  difficult  at  best 

•  Identifying  future  competencies  to  ensure  most  relevant  training  content 
Human  Capital 

•  Changing  workforce  demographics  requiring  newer  methods  of  training 
and  management 

Client  Business  Environment 

•  Achieving  acquisition  excellence  in  a  fiscally  constrained  environment 
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External  Forces 


Source:  DAU 
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Years  of  Service 

Traditional  Generation  —  Baby  Boomers  “Generation  X  Generation  Y  “Millenium 

(Before  1946)  (1946-1964)  (1965-1976)  (1977-1989)  (1990  -  Present) 


Bimodal  Demographics 
(Space  Industry) 


Area  of  Concern 

- A - , 


54%  of  the  S&T  Workforce  is 
Over  45  and  33%  wifi  be  retirement 
eligible  in  5  years 


Professional  Growth 
vs.  Time _ ^ 


cR 
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Source:  LMSC 


Level 

4 


Source:  DAU 


Program  Systems  Engineer  (Master) 
(minimum  1 0  years  of  SE  experience) 


Level  3  -  Systems 
Engineer(Expert) 

(minimum  6  years  of 
SE  experience) 

Level  2  -  Acquisition  Engineer 
(Journeyman) 

(minimum  2  years  of  experience) 

Level  1  -  Acquisition  Engineer 
(Entry) 

(minimum  1  year  of  experience) 


SPRDE/Systems  Engineering  Career  Field  SOUTCG'  DAU 
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Technology  Capability 


Technological:  Acceleration  of  Innovation  in  the 
21st  Century  -  Facilitating  Our  Ability  to  Build 
Move  Complex  Systems 


of  New  Technological 
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Innovation  is 


Doubling 


Upfront 


SE/SW 


Requires  More 


Engineering  to  Leverage  Trends 
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Linear  vs. 

Exponential  Growth: 


Linear  Plot 
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Technological:  Augustine’s  Law  Holding  -  Growth  of 
Software  is  an  Order  of  Magnitude  Every  10  Years 
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Technological:  Moore's  Law  Holding  -  The  Number  of 
Transistors  That  Can  be  Placed  on  an  Integrated 
Circuit  is  Doubling  Approximately  Every  Two  Years 


Integrated  Circuit  Complexity 
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Source:  Intel 


Technological:  Increasing  Rate  of  Adoption 

Electricity 
(1873) 


Automobile 
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Telephone 
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Cell  phone  =  14  years 
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Source:  Rich  Kaplan,  Microso, . 
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Human  Capital:  Refocusing  University  Curriculums  - 
Alignment  of  Software  Systems  Engineering 


System 

Analysis 


System 

Testing 


Systems  Engr. 


System 

Design 


Systems 

Engineering 


SW  Systems  Engr. 
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Requirements 
Analysis _ 


SW  Systems 
Engineering 


Architectural 
SW  Design 


Systems  Engr. 


SW  System 
Testing 

_  SW  Systems  Engr. 


SW  Integration 
Testing 


SW  Engineering 


Detailed  SW 
Design 


SW  Engineering 


SW  =  Software 


Code  and 
Unit  Test 


OSD  Initiatives:  Graduate  Software  Engineering  Reference  Curriculum  (GSwERC) 
&  Body  of  Knowledge  and  Curriculum  to  Advance  Systems  Engineering  (BKCASE) 
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Human  Capital:  Using  Core  Competencies 


+  Accurate  identification  of  required  competencies  are 
important  to  support  the  curriculum  review  and 
development  effort  needed  to  ensure  the  best  and  most 
relevant  training. 


■ 

Software  Engineer  III 

r 

Software  Engineer  II 

i 

Software  Engineer  1 

r 

Knowledge  Application  domain 

Procedural  design 
Cobol  &  Assembler 
Numerical  analysis 

Skills  Requirements  analysis 

System  design 
Project  management 
Debugging 

Process  Abilities  Integrated  team  design 
Fagan  inspections 
Test  procedures 
Change  control 


Competency  Family 
Software  Engineering 


Current  Resource  Profile  (initial  inventory) 


Workforce 

Staffing  by  Capacity  Lc 

jvel 

Competency 

1 

II 

III 

Software  Engineer 

17 

25 

12 

User  Training 

2 

8 

4 

Current  Resource  Needs  (one-year  cycle) 

Workforce 

Current  Staffing  Level  Nc 

ieded 

Competency 

i 

II 

III 

Software  Engineer 

23 

30 

15 

User  Training 

4 

9 

6 

Strategic  Workforce  Needs  (2-5  year) 

Workforce 

2010Staffing  Level  Nee 

ded 

Competency 

1 

II 

III 

Software  Engineer 

31 

35 

18 

User  Training 

4 

10 

8 

12th  Annual  NDIA  Systems  Engineering 

Software  Engineering  Institute  Carnegie  Mellon  Dr.  Kenneth  e.  N^mer,  October  2009  13 

©2009  Carnegie  Mellon  University 


Human  Capital:  Changing  Demographics 


Demographics  of  workforce  are  changing  and  different  views  may 
emerge  with  four  generations  to  consider 

Generation  Y  professionals  entering  workforce  will  likely  necessitate 
non-traditional  training  techniques,  such  as  virtual  approaches 


Silent  Generation 
1928-1945 


Generation  X 
1965-1980 


Generation  Y/Millenials 
1981-2000 


Baby  Boomers 
1946-1964 


Hard  worker 
Respects  authority 
Work  is  obligation 
Formal  communicator 
Work/family  separation 


Workaholic 
Questions  authority 
Works  efficiently 
Competitive 
Little  work/life  balance 


Technically  savvy 
Embraces  diversity 
Requires  supervision 
Indirect/virtual  communicator 
Demands  work/life  balance 


Technically  advanced 
Prefers  informality 
Needs  structure  and  direction 
Direct/immediate  communicator 
Seeks  work/life  balance 
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Client  Business  Environment:  Increasingly  Complex 


Characteristics 


Commercial 
Software  Products 


Information  Technology  & 
Internet  Financial  Services 


Government  Aerospace 
Systems 


Market 


Commercial 


Information  technology  & 
internet 


Government 


Industry 


Software 


Financial 


Aerospace 


Packaging 


Products 


Services 


Systems 


Primary  Output 


Software 


Integrated  system  engr  & 
HW&  SW  &  network 


Integrated  system  engr  & 
HW&  SW  &  network 


Purpose 


User  empowerment: 
effectiveness, 
efficiency,  creativity 


Organization/business 

operations 


Mission/science 

capabilities 


Project  Duration 


1-36  months 


1-18  months 


6  months  -  10  years 


Team  Size 


1-1000's 


1-1 000's 


1 0's-1 000's 


Ratio  of  Custom 
to  COTS/Reuse 


Software:  Low-high 


Business  logic:  High 
Others:  Low 


All:  High 


Agreement 


License 


Service  level  agreement 


Contract 


Customer 


External 


Internal  and  external 


External 


#  Customers 


100's-l, 000, 000's 


1-1. 000,000's 


1 


Focus 


Features,  Time-to- 
market,  Ship  it 


User  experience.  Workflow 
cycletime.  Uptime 


Reliability,  Milestones, 
Interdependencies 


Source  -  Northrop  Grumman 
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Client  Business  Environment:  Acquisition  Shifts 


System 

Operation 
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“Acqi 

lisition 

1 

r 

2005  study  confirmed*: 

•  In  advanced  knowledge-based  organizations,  management’s 
desire  for  the  flow  of  knowledge  is  greater  than  the  desire  to 
control  boundaries 

•  Unlike  the  matrix  organization,  there  is  less  impact  on  the 
dynamics  of  formal  power  and  control 

Using  Communities  of  Practice  to  Drive  Organizational  Performance  and  Innovation,  2005,  APQ 
study 

Advanced  Knowledge-Based  Organizations  (Big  A) 


System 

Construction 


-Operational 


Constructive 


System 

Operation- 


System 
Conslmctjpn 


Program 

Management 


-Programmatic  \ 


Program 

Management 


From  “Science  and  Technology  to  Support  FORCEnef,”  Raytheon  TD-06-008. 

“acquisition”  Used by permission 


Ref:  Jim  Smith,  (703)  908-8221 , ids@sei.cmu.edu 
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Systems  Engineering  Approach 


Phase  1 

Identify/Collect  Data 
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Identify  Training 
Courses 
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Identify/Select 
Reference  Model 
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Training  Process 


Review  Legacy/ 
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Phase  2 
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Selected  based  on 

amount/type  of  data  to  be  reviewed 
availability  of  a  reference  model 
requirements,  logical  and  physical  loops 
iteration  and  recurrision  activities 
access  to  key  stakeholders 


Training  Class 
Coverage  Gaps 


Phase  3 

Formulate/Codify  Findings 
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Findings,  Impacts, 
Recommendations 


Write  Draft  Report 


Phase  4 

Develop/Deliver  Results 
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Project  Objectives 

During  assessment  Phase  1  project  objectives  were 
formulated  in  terms  of  five  questions 

•  Do  coverage  gaps  exist  in  the  training  of  acquisition  best 
practices? 

•  Do  gaps  exist  in  acquisition  training  on  the  unique  aspects 
of  the  client’s  system  acquisitions? 

•  Do  gaps  exist  in  the  training  of  the  client’s  acquisition 
lifecycle  framework  and  processes? 

•  Do  best-practice  gaps  exist  in  the  client’s  organizational 
training  processes? 

•  Do  gaps  exist  in  identifying  training  requirements  for 
satisfying  the  acquisition  workforce  core  competencies? 
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Reference  Model 


Evaluated  client  s  acquisition  training  program  components 
using  Capability  Maturity  Model  Integration®  for  Acquisition 
(CM Ml®  -ACQ)  as  reference  model 
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Assessment  Framework:  CMMI®-ACQ 


Operational  Need 


Focus  on  Acquisition  Best  Practices 

(Acquirer) 


Representative  Results:  Question  1 


Question  1:  Do  Coverage  Gaps  Exist  in  the  Training  of  Acquisition  Best  Practices? 

Findings: 

Recommendations: 

■  Detailed  findings  awaiting  client 
approval 

■  Conducting  a  review  to  assess  use  of  web- 
based  and  non-traditional  acquisition  training 

Impacts: 

Considerations: 

■  Missing  opportunities  to 
~  attract  more  students 
~  provide  training  on  the  most  relevant 
issues 

Consider:  Leveraging  of  efforts  by  DAU, 
commercial  industry  and  academia 

■  Conducting  a  review  of  best  practices 
for  e-learning 

~  effectively  plan 
~  save  resources 

~  provide  a  richer  variety  of  courses 
~  continuously  improve  training  processes 

Consider:  Using  DAU’s  Acquisition  Best 
Practices 

■  Making  a  better  use  of  repository 
information 
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Lessons  Learned 


Tsunami 


•  Tsunami-like  impacts  on  new  acquisition 
training  requirements 

•  Rapid,  large-scale  disturbance  of 
current  training  needs  envisioned 

•  Forces  will  include  technological, 
human  capital,  external  and  government 
needs 

•  Training  departments  have  incorporated 
best  acquisition  practices  into  their  training 
courses;  however 

•  Mapping  of  core  competencies  to 
training  courses  needs  to  be  done 

•  Training  architectures  needed 

•  Developers  of  organizational  training 
processes  could  benefit  from  the  application 
of  systems  engineering 


Images  of  the  Ocean  Floor 
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Wrap  Up 


Software  Engineering  Institute 
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Contact  Information 

Dr.  Kenneth  E.  Nidiffer,  Director  of  Strategic  Plans  for  Government 
Programs 

Software  Engineering  Institute,  Carnegie  Mellon  University 
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Fax:  +  1  703-908-931 7 

Email:  nidiffer@sei.cmu.edu 
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Using  Simulation  to  Define  and  allocate 
probabilistic  Requirements 
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•  General  thesis 

•  Integration  of  model-based  system  engineering 
with  simulation  in  3D  synthetic  environments  is  a 
cost  effective  way  to  design  and  analyze  systems 

•  This  is  illustrated  by 

•  analyzing  the  feasibility  of  a  proposed  design  to 
add  terrain  following  and  obstacle  avoidance  to  an 
existing  air  vehicle 

•  Objective 

•  Improve  requirements  by  adding  bounds  to 
requirements  based  on  analysis 


Engineering  Application  Example 


•  Problem 

•  Determine  if  the  proposed  design  solution  will 
meet  customer  needs 


•  Work  to  be  done 

•  Refine  requirements  to  make  them  bounded  and 
precise 

•  Determine  critical  environmental  issues 

•  Prototype  design  sufficiently  to  verify  feasibility 

•  Validate  refined  requirement  with  customer 
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Initial  Requirement  For 
Upgrade 

•  Initial  requirement 

•  avoid  terrain  and  obstacles 
while  flying  low  over 
mountainous  terrain  to 
avoid  detection 

•  Proposed  design 

•  specific  radar  integrated 
into  the  avionics  system 


Approach: 


Environment 


•  Integrated  radar  data  with  avionics 

•  Performance  model  of  aircraft 

•  Climb  rate 

•  Velocity 

•  Sensor  models 

•  Detection  distance 

•  Sweep  pattern 

•  Environment 

•  Mountains 

•  Obstacles 

•  Weather  4 
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Approach:  Integrated  Executable 
Avionics  Model  With  Simulation 
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Approach:  Model  System  and 
Environment 


•  Simulation  developed  to  model  the  characteristics  of 
the  system  and  environment 

•  Run  simulation  multiple  times  varying  all  of  the  inputs 


Collect  results  for  statistical  analysis 


System 

Attributes 


Aircraft  speed 
Aircraft  climb  rates 
Radar  power 
Radar  scan  rates 
Radar  sweep  pattern 


Simulation 


Environmental 

Conditions 


r 

Obstacle  Height 
Obstacle  Location 
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Scenarios  -  Terrain  Variation 


•  Obstacle  at  height  of 
max  elevation  range 


•  Obstacle  at  half  height 
of  max  elevation  range 


20  deg 


•  Plateau  at  max 
elevation  range 

•  Represents  Full 
sweep 
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Scenarios  -  Scan 


•Top  of  the  right  bar 

•  Bottom  of  the  right  bar 

•  Top  of  middle  bar 

•  Bottom  of  middle  bar 

•  Top  of  left  bar 

•  Bottom  of  left  bar 
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Position 


•  2  scan  bars 
•  Faster 


•  3  scan  bars 
•  More  detail 


Scenari 


Rate  Variation 


•  Normal  rates 


•  Double  up  rate 


•  Double  down  rate 


•  Half  turnaround 
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•  Velocity  held  constant 

•  2  bars,  normal  pattern, 
normal  rates 


Detection  Distance  versus  time  for  5  runs 


>  l 


♦  >  A 


•  ■  ►  A 


Variable 

•  Baselhe 
■  full  sweep 

♦  3  bars 
A  no  pause 
►  double  rate 


3  bars,  normal  pattern 
normal  rates 

2  bars,  normal  pattern 
double  rates 


line 


2  bars,  full  pattern,  normal 
rates 

2  bars,  no  pauses  for  the 
sensor  to  turn  around, 
normal  rates  a 


•  Orange  (Double  scan  rate)  and 
Blue  (Faster  turnaround  time)  runs 
lead  the  others 

•  Red  line  (Full  sweep)  is  slowest 

•  No  difference  between  Black 

(baseline)  and  Green  (more  scan 
bars)  12 
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•  Running  the  simulation 

•  Showed  what  impacts  the  detection  distance  which  in 
turn  impacts  the  time  budget 

•  Showed  the  behavior  of  detection  distance 

•  Can  put  results  and  observation  together  to  refine 
allocated  detection  requirements 

•  Flight  path  impacted  by 

•  Airspeed 

•  Climb  rate 

•  Detection  distance 

•  Obstacle  height 

•  Sweep^pat' 
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Closer  to  a  bounded  requirement 


•  Aircraft  shall  compute  a  safe  flight  path  within  TBD 
seconds  of  detecting  obstacles  of  TBD  milliradians  in 
size  while  flying  level  at  an  airspeed  between  TBD 
and  TBD  knots  with  an  ability  to  climb  at  the  rate  of 
at  least  2000  feet  per  minute. 
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Analysis  of  Parameter 


•  Use  physics  to  form  equation  with  parameters  that 
impact  the  time  budget  for  safe  flight  computations 
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•  Analyze  each  of  the  inputs  to  determine  their 
distribution,  mean  and  standard  deviations. 

•  E.g.  Run  simulations  on  the  input  detection  distance 


Simulation 

results 


A 


Analysis 


Distribution,  mean, 
standard  deviation 
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Sensitivity  Analysis  To  Determine 
Probability  of  Non  Compliance 


•  Run  a  sensitivity 
analysis  to  determine 
the  probability  of 
compliance 


Climb  rate 


Airspeed 

■  + 


Obstacle  Height 


Detection  Distance 


time  budget  (t) 


o 

c 

0) 
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-10.96 


2.32 


15.60 


28.89 


42.17 


55.46 
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•  From:  Avoid  terrain  and  obstacles  while  flying  low 
over  mountainous  terrain. 

•  To:  Aircraft  shall  compute  a  safe  flight  path  99.98% 
of  the  time  within  0.1 1  seconds  of  detecting 
obstacles  of  0.5  milliradians  in  size  while  flying  level 
at  an  airspeed  between  200  and  280  knots  with  an 
ability  to  climb  at  the  rate  of  at  least  2000  feet  per 
minute. 
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•  Simulations  help  clarify  environmental  impacts 

•  Physics  and  geometry  used  to  determine  model 

•  Simulation  with  statistical  analysis  used  to  determine 
distribution  and  mean 

•  Sensitivity  analysis  to  determine  PNC 
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A  Systems  Engineering 
Approach  to  Multi-Level 
Security  in  a  Service 
Oriented  Architecture 

Tim  Greer 

Principal  Systems  Engineer 
301-788-4882'' 
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Presentation  Overview 


•  Definitions 

•  Architecture  Approach 

-  Requirements  Analysis 

-  Security  Layers  -  OEM  Layers 

-  Threats  and  Countermeasures 

•  Design  Considerations 

•  Performance  Considerations 

•  Cost  Considerations 

•  Conclusion 
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Definitions 


•  Multi-Level  Security  (MLS)  VS  Multiple  Security  Levels 
(MSL) 

-  MLS  -  Data  from  different  security  classification  levels  on  the 
screen  at  the  same  time  or 

-  MLS  -  Data  from  different  security  classification  levels  or 
releasability  restrictions  stored  in  the  same  data  base 

-  MSL  -  Multiple  security  enclaves  co-located  but  physically 
separated 

-  MSL  -  Data  from  only  one  security  enclave  on  a  screen  at  a  time 
-  KVM  switch  may  connect  to  workstation  to  multiple  security 
enclaves  -  but  each  must  be  logged  into  separately 


Multiple  Security  Levels  (MSL) 


WC-NDC 


SECMg 

LSIPRN 


Nslwwk 

CotlM 
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One-way 

Transfer 


OWT  <  OWT 


Multi-Level  Security  -  Definition  2 
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Multi-Level  Security  -  Definition  1 


Flexible  security  operation  based  on  changing  Labels  or  Policies 


SCI  User 
Secret  User 
Unclass  User 


Definitions 


•  Service  Oriented  Architecture 

-  A  standards-based  architectural  paradigm  that  enables  mission 
processes  through  discovery  and  invocation  of  published, 
shared,  discrete,  and  reusable  mission  and  infrastructure 
services  across  a  network 


-  Designed  to  allow  a  community  of  service  providers  and 
consumers  to  achieve  value  by  aligning  services  to  mission 
processes  and  enabling  better  mission  agility 

-  Services  are  published,  discoverable,  invoked,  and  consumed 

-  Services  may  be  discovered  and  consumed  either  internally  or 
externally  to  an  enterprise 


-  Services  are  designed  to  be  predominantly  loosely  coupled 
however  a  family  of  services  may  be  built  and  designed  to  work 
together 
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Definitions 


•  Authentication  -  Establishes,  verifies,  and  identifies  a 
person  or  a  process  -  includes  identity  assertion. 

•  Authorization  -  The  process  of  determining,  by 
evaluating  applicable  access  control  information, 
whether  a  subject  is  allowed  to  have  the  specified  types 
of  access  to  a  particular  resource. 

•  Role  Based  Access  Control  (RBAC)  -  The  process  of 
restricting  access  to  a  service  resource  based  on  the 
roles  associated  with  the  consumer  log  in. 


Requirements  Analysis 


-  The  following  information  must  be  collected  prior  to 
contacting  the  Designated  Accreditation  Authority  (DAA) 

•  The  category,  classification,  and  all  applicable  security  markings 
for  all  of  the  information  on,  or  to  be  put  on,  the  system; 

•  The  need-to-know  status  of  the  users  on  the  system,  including 
their  formal  access  approval(s),  dearance(s),  and 
nationality(ies); 

•  The  perimeter  and  boundary  of  the  system; 

•  The  operating  environment  of  the  system  and  connecting 
systems,  including  the  service  provided  (e.g.,  electronic  mail, 
Internet  access),  and  foreign  access  to  the  system,  connecting 
systems,  and  the  facilities  housing  these  systems;  and 

•  The  technical  and  administrative  security  requirements  of  the 
system. 


Architecture  -  Requirements  Analysis 


•  Security  Requirements  are  often  not  explicitly  stated 

-  Look  for: 

•  Data  transfer  requirements 

•  Access  to  a  particular  network  or  the  internet  requirements 

•  Visualization  of  data  requirements 

•  Reference  to  a  directive  or  standard  requirements 

•  Connection  to  applications  or  systems  (interoperability) 
requirements 

-  When  connecting  to  networks  like  SIPRnet,  JWICS,  NGAnet, 
NSAnet,  DIAnet,  NIPRnet,  CENTRIX 

•  Contact  the  Designated  Accreditation  Authority  (DAA) 

•  Obtain  the  appropriate  STIGS,  SNAC  Guides,  DCID  6/3,  MAC 
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Dissemination  Requirements 


*  DCID  8  States: 

-  Maximize  Production  of  Intelligence  at  Multiple  Security 
Levels. 

•  Write-to-Release 

•  Tearlines 

•  Content  Management 

•  Data  Tagging 

*  ICD  501  States: 


-  1C  elements  shall  have  a  predominant  responsibility  to: 

•  Provide 

•  Discover 

•  Request  relevant  information 
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Requirements  Analysis 


•  DIACAP  Training  - 

http://iase.disa.mil/eta/diacap/diacap1/index.htm 

•  DCID  and  ICD  guides 
http://www.fas.org/irp/offdocs/dcid.htm 

•  DCID  6/3  Online  Manual 

http://www.fas.org/irp/offdocs/DCID  6-3  20Manual.htm 

•  Intelligence  Community  Directive  Number  501 
www.  dni.  aov/electronic  reading  rgom/ICD  501.pdf 

•  DoD  Metadata  Specification 
https://metadata.dod.mil/mdr/irs/DDMS/ 


Threats  &  Countermeasures 


Securing  the  Application 


Input  validation 
Authentic  alt  ion 
Authorization 

Configuration  Management 

Sensillue  Data 


Session  Manager  enl 

Cryptography 
Parameter  Manipulation 
Exception  Management 

Auti ting  and  Logging 


wet» 

Server 


Application 

Server 


Server 


Securing  the 
Network 
Router 
FireivaJI 
Swrlch 


Patches  and 

Securing  the  Host 

Accounts 

Potts 

Updates 

Services- 

Protocols 

Files  and  Directories 

Registry 

Shares 

Auditing  and  Logging 
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Threats  and  Countermeasures 


Anatomy  of  a  Threat 


Survey  and 
Assess 


Exploit  and 
Penetrate 


EwsiMfl  PrrvilflDfl^ 
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•  Spoofing 

•  Tampering 

•  Repudiation 

•  Information  disclosure  -  Sniffing 

•  Denial  of  service 

•  Elevation  of  privileges 

•  Session  Hijacking 


Countermeasures 


*  Spoofing  -  Strong  Authentication  (Cetificates)  -  Mutual 
Authentication  -  SSL  -  Encryption 

*  Tampering  -  Strong  Authorization  -  Data  Hashing  -  Digital 
Signatures  -  Message  Validation  Protocols 

*  Repudiation  -  Secure  Audit  Logs  -  Digital  Signatures 

*  Information  disclosure  -  Sniffing  -  Strong  Encryption  -  SSL 

*  Denial  of  service  -  Intrusion  Detection  System  (IDS)  - 
Defense  in  Depth  -  Buffering  and  Resource  Throttling 
Techniques  -  Validate  &  Filter  Input 

*  Elevation  of  privilege  -  XML  Gateway  -  Use  Least 
Privileged  User  Accounts 

*  Session  Hijacking  -  Strong  Encryption  -  Timestamp 
Synchronization  &  Re-authentication 


Design  Considerations  -  Direct 
Authentication 


Validate 


(?)  Credential 


Client 


Service 


Identity  Store 
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Design  Considerations  -  Brokered 
Authentication  -  Mutual  Authentication 


•  Kerberos  Ticket 

•  X.509  PKI  Certifi 

•  STS 


G 

Ideality  Store 


(5)  Validate 
Token 
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Message  Layer  VS  Transport  Layer 
Security 


•  Brokered  Authentication  can  be  implemented  at  the 
Message  Layer  or  Transport  Layer 

-  Message  Layer  Security  provides  for 

•  Data  Confidentiality 

•  Data  Origin  Authentication 

•  Data  Integrity 

-  Message  Layer  Security  is  more  complex 

-  Transport  Layer  Security  provides  for 

•  Minimal  code  and  configuration  work 

•  With  Kerberos  can  work  across  multiple  system  hops 

-  Transport  Layer  is  simpler  -  but  does  not  provide  Data  Integrity  - 
should  be  used  with  SSL 

•  SSL  can  only  be  used  point  to  point  VS  end  to  end 


Policy  Enforcement 


External 

Trusted 

Authority 


Jystem 

Bounda 


Internal 

User 


External 
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XACML 
SAML 
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PEP  =  Policy 
Enforcement  Point 
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User 

Account 

Credentials 


Data 


Policy  Enforcement 


Policy  Enforcement 


Trusted  Subsystem 


Jystem 

Boundary 


Web  Service  1 


Web  Service  2 
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•  Ensure  all  audit  records  include  date  and  time  of  action,  the  system 
locale  of  the  action,  the  system  entity  that  initiated  or  completed  the 
action,  the  resources  involved,  the  action  involved,  and  successful  and 
unsuccessful  logons  and  logoffs. 

•  Protect  the  contents  of  audit  trails  against  unauthorized  access, 
modification,  or  deletion. 

•  Maintain  collected  audit  data  at  least  5  years  and  review  at  least  weekly. 

•  Maintain  an  audit  trail  that  includes  selected  records  of:  Accesses  to 
security-relevant  objects  and  directories,  including  opens,  closes, 
modifications,  and  deletions. 

•  Maintain  an  audit  trail  that  includes  activities  at  the  system  console 
(either  physical  or  logical  consoles),  and  other  system-level  accesses 
by  privileged  users. 

-  Individual  accountability  (i.e.,  unique  identification  of  each  user  and 
association  of  that  identity  with  all  auditable  actions  taken  by  that 
individual). 

-  Periodic  testing  by  the  ISSO  or  ISSM  of  the  security  posture  of  the  IS  by 
employing  various  intrusion/attack  detection  and  monitoring  tools. 


Cost  and  Performance 


Criteria 

Definition 

Rank 

Weight 

Standard  Evaluation  Criteria 

Cost 

Includes  both  the  recurring  and  non-recurring  costs. 
Include  labor,  resources  and  any  lifecycle  charges. 

5  -  Very  Low  Cost  Impact 

4  -  Low  Cost  Impact 

3  -  Medium  Cost  Impact 

2  -  High  Cost  Impact 

1  -  Very  High  Cost  Impact 

15% 

Meets  Requirements 

Indicates  the  ability  of  a  solution  to  fully  and  or  partially 
meet  the  requirements  defined  in  the  A-specification 
and  B-Specification 

1  -  Very  Low  Meets  Requirements 

2  -  Low  Meets  Requirements 

3  -  Medium  Meets  Requirements 

4  -  High  Meets  Requirements 

5  -  Very  High  Meets  Requirements 

20% 

Install  Base 

Defines  how  widely  used  a  solution  is  and  how  many 
users  may  be  trained  on  the  solution  today.  Not 
specifically  meant  to  portray  commercial  use,  it  also 
includes  GOTS  standards  that  have  been  adopted  by 
gov't  agencies. 

1  -  Very  Low  Install  Base 

2  -  Low  Install  Base 

3  -  Medium  Install  Base 

4  -  High  Install  Base 

5  -  Very  High  Install  Base 

5% 

Performance 

Indicates  the  speed  and  quality  at  which  a  solution 
executes  its  functions.  If  possible,  it  should  be  based 
on  hard  execution  data.  If  this  is  not  feasible,  the 
measure  can  be  based  on  the  architectural  choice 
made  that  may  enhance  or  impede  performance.  It 
also  considers  the  consistency  of  the  performance  for 
all  the  users 

1  -  Very  Low  Performance 

2  -  Low  Performance 

3  -  Medium  Performance 

4  -  High  Performance 

5  -  Very  High  Performance 

10% 
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Cost  and  Performance 


Criteria 

Definition  Rank 

Standard  Evaluation  Criteria 

Dependencies 

Defines  the  number  of  strict  needs  the 
solution  requires  to  operate.  These 
dependencies  can  be  at  the  infrastructure 
level  or  at  the  system  level.  Should  be  an 
indicator  of  how  easy  it  will  be  to  integrate 
the  solution. 

5  -  Very  Low  Dependencies 

4  -  Low  Dependencies 

3  -  Medium  Dependencies 

2  -  High  Dependencies 

1  -  Very  High  Dependencies 

5% 

Certification  &  Accreditation 

Indicates  if  the  solution  has  been  accredited 
previously.  If  not,  it  should  provide  some 
measure  that  indicates  if  it  would  be  easily 
accredited  based  on  similar  products,  its 
lifecycle,  its  implementation  etc... 

1  -  Very  Low  Certification  & 
Accreditation 

2  -  Low  Certification  &  Accreditation 

3  -  Medium  Certification  & 
Accreditation 

4  -  High  Certification  &  Accreditation 

5  -  Very  High  Certification  & 
Accreditation 

5% 

Interoperability 

Defines  the  solution's  capability  to 
interoperate  with  diverse  systems  and 
infrastructure  capabilities.  Indicates  if  the 
solution  is  based  on  open  (non-proprietary) 
standards,  that  it  has  exposed  interfaces  and 
is  adaptable  to  many  environments.  This 
also  includes  the  product's  ability  to  operate 
in  service  oriented  environment. 

1  -  Very  Low  Interoperability 

2  -  Low  Interoperability 

3  -  Medium  Interoperability 

4  -  High  Interoperability 

5  -  Very  High  Interoperability 

5% 

Reliability 

Measures  a  solution's  ability  of  a  system  to 
perform  and  maintain  its  functions  in  routine 
circumstances,  as  well  as  hostile  or 
unexpected  circumstances.  Systems  with  no 
track  record  or  with  complex,  unreliable 
software  will  probably  score  lower. 

1  -  Very  Low  Reliability 

2  -  Low  Reliability 

3  -  Medium  Reliability 

4  -  High  Reliability 

5  -  Very  High  Reliability 

5% 

DtFtNSt 


Cost  and  Performance 


Criteria 

Definition  Rank  Weight 

Standard  Evaluation  Criteria 

Manageability 

Requires  the  product  to  be  capable  of  being  managed  in  a 
production 

1  -  Very  Low  Manageability 

2  -  Low  Manageability 

3  -  Medium  Manageability 

4  -  High  Manageability 

5  -  Very  High  Manageability 

5% 

Scalability 

Defines  a  products  capability  to  add  additional  hardware  or 
software  to  the  system  for  additional  load  (i.e.  additional 
users,  messages,  clustering). 

1  -  Very  Low  Scalability 

2  -  Low  Scalability 

3  -  Medium  Scalability 

4  -  High  Scalability 

5  -  Very  High  Scalability 

5% 

SWAP  Impact 

Defines  the  impact  on  the  existing  deployed  system  for 
space,  weight  and  power  availability. 

5  -  Very  Low  Hardware  Impact 

4  -  Low  Hardware  Impact 

3-  Medium  Hardware  Impact 

2  -  High  Hardware  Impact 

1  -  Very  High  Hardware  Impact 

15% 

Flexibility 

Defines  the  additional  capabilities  the  product  adds  to  the 
trade  over  and  above  the  basic  requirements  for  solutions 
to  other  complexities  of  the  system  (i.e.  real  time  changing 
of  logging  levels,  monitoring  of  metrics,  debugging  of 
service  transactions,  configurable) 

1  -  Very  Low  Intelligence  Community  Standard 

2  -  Low  Intelligence  Community  Standard 

3  -  Medium  Intelligence  Community  Standard 

4  -  High  Intelligence  Community  Standard 

5  -  Very  High  Intelligence  Community  Standard 

2.5% 

Intelligence 

Community  Standard 

Defines  the  compliance  with  DCID  6/3. 

1  -  Very  Low  Intelligence  Community  Standard 

2  -  Low  Intelligence  Community  Standard 

3  -  Medium  Intelligence  Community  Standard 

4  -  High  Intelligence  Community  Standard 

5  -  Very  High  Intelligence  Community  Standard 

2.5% 

DEFENSE 


on  Tomorrow 


Gartner’s  Magic  Quadrant 


Focus  on  Tomorrow 


Ability  to  Execute 

(In  Technology, 
visibility,  services, 
features) 


IS&GS 

DEFENSE 


Challengers 


Leaders 


Execute  well  today  or 
may  dominate  a  large 
segment,  but  does  not 
yet  understand  market 
direction 

Executes  well 
today  and  is 
well-positioned 
for  tomorrow 

Focuses  successfully  on 
a  small  segment,  or  is 
unfocused  and  does  not 
out  innovate  or 
outperform  others 

Understands 
where  the 
market  is  going 
or  has  a  vision 
for  changing 
market  rules,  but 
does  not  yet 
execute  well 

Niche  Players 


Visionaries 


Accreditability 


•  Evaluation  by  independent  Laboratories  -  Common 
Criteria 

http://www.commoncriteriaportal.org/ 

•  Early  Engagement  of  DAA 

•  Thorough  testing  using  the  STIGS,  SNAC  Guides  and 
other  guiding  documents 

•  Well  documented  architecture 

•  Well  documented  system  and  operational  procedures 


Conclusion 


•  Engage  DAA  Early 

•  Requirements  Analysis  early  and  complete 

•  Identify  Threats 

•  Determine  Countermeasures 

•  Evaluate  Architecture  Alternatives 

•  Balance  Cost,  Performance,  Security  through  Analysis  of 
Alternatives  exercise 

•  Leverage  Existing  Capabilities  While  Implementing  New 
Technologies 


IS&GS 

DEFENSE 
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OVERVIEW 


>  Current  need  to  reduce  cost  while  improving 
performance  of  Legacy  Systems 


>  Definition  of  Classic  Systems  Engineering 
Process 

>  Example  application  and  Case  Study 

>  Where  to  from  here? 


Systems  Engineering  10-29-09 


al^-  j  CLASSIC  SYSTEMS  ENGINEERING  PROCESS 


An  orderly  approach  to  meeting  customer 
needs  through  detailed  design. 
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POTENTIAL  APPLICATIONS 
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CASE  STUDY:  TAXICAB 


>  Company  Objective:  Increase  Profitability 

>  Approach:  Systems  Engineering 

>  Facts:  100  Vehicles,  75,000  Miles/Year/Vehicle 

200  Drivers,  Management,  Dispatch, 
Advertising,  Health  Insurance,  Auto 
Insurance,  Vehicle  Support  (Repair, 
Fuel,  etc.) 
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REQUIREMENTS 


OBEY 

CITY 

LAWS 


SOLUTION 

SPACE 


TRAVEL 

RANGE 
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BASELINE  COST  ESTIMATE 


Appropriate 
cost  elements 
for  the 
particular 
customer 


Cost  Element 

Cost  /  Year 

Total  Cost 

$10,500,000 

Total  Labor 

$4,340,000 

Driver 

$3,200,000 

Dispatch 

$140,000 

Management 

$1,000,000 

Fuel 

$1 ,500,000 

Maintenance 

$1 ,200,000 

Vehicles 

$700,000 

Equipment 

$500,000 

Insurance 

$1 ,1 56,000 

Health 

$856,000 

Auto 

$300,000 

Vehicle  Replacement 

$500,000 

Advertising 

$52,000 

Miscellaneous  Cost 

(Rent,  Utilities,  etc.) 

$1 ,752,000 

Systems  Engineering  10-29-09 


ALTERNATIVES  BY  FUNCTION 


>  Support 

9  Smaller  Cars 

•  Electric  Cars 

•  Repair  Contract 

•  Run  to  Fail 

•  Etc. 

>  Insurance 

9  Self  Insured  Drivers 

9  Negotiate  with  Insurance 
Companies 

•  Hire  Lawyers 
9  Liability  Only 

•  Etc. 


>  Driver  Function 

9  Defensive  Driver  Class 

•  Work  for  Tips  Only 

•  Robotic  Driver 

•  Pull  Off  &  Wait 

•  Etc. 

>  Advertising 

9  Utilize  Car  In/Out 
9  More  Trips 
9  Double  Budget 
9  New  Media  Outlet 
9  Etc. 


Broader  analysis  provides  a  wider 
range  of  opportunities. 
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ALTERNATIVE  ANALYSIS 


Evaluation  Pareto 


O 

(0 
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Alternative 
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EVALUATION 


DOUBLE 

ADVERTISING 

BUDGET 


NEW 

MEDIA 

OUTLET 


OPTIMAL  SOLUTION 


CAR 

TYPE 


ROBOTIC 

DRIVERS 


HIRE 

LAWYERS 


REPAIR 

CONTRACT 


The  Best  solution  is  cherry  picked 
from  several  solutions. 
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RESULTS 


Systems  approach  yields  a  better  solution  than 
any  individual  fix. 


Systems  Engineering  solutions  are  typically  two  to 
three  times  better  at  meeting  objectives. 
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WHERE  TO  FROM  HERE? 


How  to  train  Systems  Engineering  practitioners 

to  work  in  legacy  world. 

Are  you  fired  up,  ready  to  go? 


Fired  up,  ready  to  go? 
Fired  up,  ready  to  go? 

Fired  up,  ready  to  go? 
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I  Discussion  Topics 


•  How  the  world  has  changed 

•  The  current  state  of  software  engineering 
education 

•  Creating  and  disseminating  a  new  reference 
curriculum 


•  And  next? 


There  are  precious  few  interesting 
man-made  systems  whose  success  is 
not  critically  dependent  on  software. 

Twenty  years  from  now,  software  people  will  be 
sitting  at  the  table  and  the  other  disciplines  will 
be  sitting  around  the  sides  of  the  room. 

Eberhardt  Rechtin ,  1993 


There  are  precious  few  interesting  software 
systems  anywhere  whose  success  is  not 
critically  dependent  on  the  developers 
practicing  good  systems  engineering. 


What  do  we  teach  for  a  master’s 
degree  in  software  engineering? 

•  The  last  effort  to  create  a  reference  curriculum  for 
graduate  software  engineering  education  was  by  the  SEI 
in  the  early  1 990s. 

•  There  are,  in  effect,  no  current  community-endorsed 
recommendations  on  what  to  teach  software  engineers  - 
nothing  that  recognizes  how  the  world  has  changed. 

•  Response:  create  a  project  to  create  a  new  reference 
curriculum  in  software  engineering 


The  Integrated  Software  and  Systems 
Engineering  Curriculum  (iSSEc)  Project 


•  Begun  in  May  2007  at  Stevens  Institute  of  Technology 

•  Sponsored  by  DoD  Director  of  Systems  and  Software 
Engineering 

•  Three  products  planned: 

1 .  A  modern  reference  curriculum  for  a  master’s  degree  in  software 
engineering  that  integrates  an  appropriate  amount  of  systems 
engineering 

2.  A  modern  reference  curriculum  for  a  master’s  degree  in  systems 
engineering  that  integrates  an  appropriate  amount  of  software 
engineering 

3.  A  truly  interdisciplinary  degree  that  is  neither  systems  nor  software 
engineering  -  it  is  both 


1st  Project  -  Graduate  Software 
Engineering  2009 

1 .  Understand  the  current  state  of  SwE  graduate 
education  (November  2007) 

2.  Create  GSwE2009  0.25  (formerly  GSwERC)  with  a 
small  team,  suitable  for  limited  review  (February  2008) 

3.  Publicize  effort  through  conferences,  papers,  website, 
etc  (continuous) 

4.  Create  GSwE2009  0.50  (formerly  GSwERC)  suitable  for 
broad  community  review  and  early  adoption  (October 
2008) 

5.  Create  GSwE2009  1 .0  suitable  for  broad  adoption 
(2009) 

6.  Transition  stewardship  to  professional  societies  (2009) 

7.  Foster  adoption  world-wide  (2009  and  beyond) 
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SWEBOK  coverage*  in  2007  across 
28  SwE  MS  programs 


*Coverage  in 
required  and  semi- 
required  courses 


The  curriculum  author  team 


•  Rick  Adcock,  Cranfield  University  and  INCOSE 
representative,  UK 

•  Edward  Alef,  General  Motors,  USA 

•  Bruce  Amato,  Department  of  Defense,  USA 

•  Mark  Ardis,  Stevens  Institute  of  Technology,  USA 

•  Larry  Bernstein,  Stevens  Institute  of  Technology,  USA 

•  Barry  Boehm,  University  of  Southern  California,  USA 

•  Pierre  Bourque,  Ecole  de  Technologie  Superieure  and  co¬ 
editor  of  2010  SWEBOK  update,  Canada 

•  John  Brackett,  Boston  University,  USA 

•  Murray  Cantor,  IBM,  USA 

•  Lillian  Cassel,  Villanova  and  ACM  representative,  USA 

•  Robert  Edson,  Analytic  Services  Inc.,  USA 

•  Richard  Fairley,  Colorado  Technical  University,  USA 

•  Dennis  Frailey,  Raytheon  and  Southern  Methodist 
University,  USA 

•  Gary  Hafen,  Lockheed  Martin  and  NDIA,  USA 

•  Thomas  Hilbum,  Embry-Riddle  Aeronautical  University, 
USA 

•  Greg  Hislop,  Drexel  University,  and  IEEE  Computer 
Society  representative,  USA 

•  David  Klappholz,  Stevens  Institute  of  Technology,  USA 

•  Philippe  Kruchten,  University  of  British  Columbia, 
Canada 


Phil  Laplante,  Pennsylvania  State  University,  Great  Valley,  USA 

Qiaoyun  (Liz)  Li,  Wuhan  University,  China 

Scott  Lucero,  Department  of  Defense,  USA 

John  McDermid,  University  of  York,  UK 

James  McDonald,  Monmouth  University,  USA 

Ernest  McDuffie,  National  Coordination  Office  for  NITRD,  USA 

Bret  Michael,  Naval  Postgraduate  School,  USA 

William  Milam,  Ford,  USA 

Ken  Nidiffer,  Software  Engineering  Institute,  USA 

ArtPyster,  Stevens  Institute  of  Technology,  USA 

Paul  Robitaille,  Lockheed  Martin,  USA 

Mary  Shaw,  Carnegie  Mellon  University,  USA 

Sarah  Sheard,  Third  Millenium  Systems,  USA 

Robert  Suritis,  IBM,  USA 

Massood  Towhidnejad,  Embry-Riddle  Aeronautical  University, 
USA 

Richard  Thayer,  California  State  University  at  Sacramento,  USA 
J.  Barrie  Thompson,  University  of  Sunderland,  UK 
Guilherme  Travassos,  Brazilian  Computer  Society,  Brazil 
Richard  Turner,  Stevens  Institute  of  Technology,  USA 
Joseph  Urban,  Texas  Tech  University,  USA 
Ricardo  Valerdi,  MIT  &  INCOSE,  USA 
David  Weiss,  Avaya,  USA 

Mary  Jane  Wiltshire,  Colorado  Technical  University,  USA 
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I  Creating  GSwE2009  0.25 


1 .  Understand  the  current  state  of  SWE  graduate 

education  (November  2007) 

2.  Create  GSwE2009  0.25  (formerly  GSwERC)  with  a 
small  team,  suitable  for  limited  review  (February  2008) 

3.  Publicize  effort  through  conferences,  papers,  website, 
etc  (continuous) 

4.  Create  GSwE2009  0.50  suitable  for  broad  community 
review  and  early  adoption  (October  2008) 

5.  Create  GSwE2009  1 .0  suitable  for  broad  adoption 
(2009) 

6.  Transition  stewardship  to  professional  societies  (2009) 

7.  Foster  adoption  world-wide  (2009  and  beyond) 


Publicize  effort 


1 .  Understand  the  current  state  of  SWE  graduate 

education  (November  2007) 

2.  Create  GSwE2009  0.25  with  a  small  team,  suitable  for 
limited  review  (February  2008) 

3.  Publicize  effort  through  conferences,  papers,  website, 
etc  (continuous) 

4.  Create  GSwE2009  0.50  suitable  for  broad  community 
review  and  early  adoption  (October  2008) 

5.  Create  GSwE2009  1 .0  suitable  for  broad  adoption 
(2009) 

6.  Transition  stewardship  to  professional  societies  (2009) 

7.  Foster  adoption  world-wide  (2009  and  beyond) 
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Publicize  effort 


1 .  Past  and  planned  presentations  and  workshops  at 
numerous  conferences,  including: 

-  NDIA  Systems  Engineering  Conferences  2007,  2008,  and  2009; 
INCOSE  International  Symposium  2008  and  2009,  ASEE  2008, 
Asian-Pacific  INCOSE  Conference  2008,  SIGCSE  2008  and 
2009,  ICSE  2009,  CSEET  2009,  ... 

2.  Short  articles  and  announcements  in  SEWORLD, 
INCOSE  Insight,  ... 

3.  Full  article  on  survey  of  existing  programs  to  appear  in 
IEEE  Software  in  fall  2009 

4.  Website  at  www.GSwE2009.org 

5.  Additional  full  articles  in  IEEE  and  ACM  magazines 
planned 
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Creating  GSwE2009  0.50  and 


1.0 


1 .  Understand  the  current  state  of  SWE  graduate 

education  (November  2007) 

2.  Create  GSwE2009  0.25  with  a  small  team,  suitable  for 
limited  review  (February  2008) 

3.  Publicize  effort  through  conferences,  papers,  website, 
etc  (continuous) 

4.  Create  GSwE2009  0.50  (formerly  GSwERC)  suitable  for 
broad  community  review  and  early  adoption  (October 
2008) 

5.  Create  GSwE2009  1 .0  suitable  for  broad  adoption 
(2009) 

6.  Transition  stewardship  to  professional  societies  (2009) 

7.  Foster  adoption  world-wide  (2009  and  beyond)  12 


Creating  GSwE2009  0.50  and 


1.0 


1 .  Understand  the  current  state  of  SWE  graduate 

education  (November  2007) 

2.  Create  GSwE2009  0.25  with  a  small  team,  suitable  for 
limited  review  (February  2008) 

3.  Publicize  effort  through  conferences,  papers,  website, 
etc  (continuous) 

4.  Create  GSwE2009  0.50  (formerly  GSwERC)  suitable  for 
broad  community  review  and  early  adoption  (October 
2008) 

5.  Create  GSwE2009  1 .0  suitable  for  broad  adoption 
(2009) 

6.  Transition  stewardship  to  professional  societies  (2009) 

7.  Foster  adoption  world-wide  (2009  and  beyond)  13 


Expectations  at  entry  (from  version  0.5+) 


DEGREE: 

The  equivalent  of  an  undergraduate  degree  in 
computing  or  an  undergraduate  degree  in  an 
engineering  or  scientific  field  and  a  minor  in  computing 

SWE  COURSE: 

The  equivalent  of  an  introductory  course  in  software 
engineering 

EXPERIENCE: 

At  least  two  years  of  practical  experience  in  some  aspect 
of  software  engineering  or  software  development 
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Outcomes  at  graduation  (from  Version 
0.5+) 

CBOK: 

Master  the  Core  Body  of  Knowledge 

DOMAIN: 

Be  able  to  apply  software  engineering  in  at  least  one  application 
domain,  such  as  finance,  medical,  transportation,  or 
telecommunications,  and  in  one  application  type,  such  as 
real-time,  embedded,  safety-critical,  or  highly  distributed 
systems.  That  ability  to  apply  software  engineering  includes 
understanding  how  differences  in  domain  and  type  manifest 
themselves  in  both  the  software  itself  and  in  their 
engineering,  and  includes  understanding  how  to  learn  a  new 
application  domain  or  type. 

DEPTH: 

Have  mastered  at  least  one  knowledge  area  or  sub-area  from 
the  Core  Body  of  Knowledge  to  at  least  the  Bloom  Synthesis i5 
level. 


Outcomes  at  graduation 


ETHICS: 

Be  able  to  make  ethical  professional  decisions  and  practice 
ethical  professional  behavior. 

SYSTEMS  ENGINEERING: 

Understand  the  relationship  between  software  engineering  and 
systems  engineering  and  be  able  to  apply  systems 
engineering  principles  and  practices  in  the  engineering  of 
software. 

TEAM: 

Be  able  to  work  effectively  as  part  of  a  team,  including  teams 
that  may  be  multinational  and  geographically  distributed,  to 
effectively  communicate  both  orally  and  in  writing,  and  to 
lead  in  one  area  of  project  development,  such  as  project 
management,  requirements  analysis,  architecture, 
construction,  or  quality  assurance. 
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Outcomes  at  graduation 


RECONCILIATION: 

Be  able  to  reconcile  conflicting  project  objectives,  finding 
acceptable  compromises  within  limitations  of  cost,  time, 
knowledge,  risk,  existing  systems,  and  organizations. 

PERSPECTIVE: 

Understand  and  appreciate  the  importance  of  feasibility  analysis, 
negotiation,  effective  work  habits,  leadership,  and  good 
communication  with  stakeholders  in  a  typical  software 
development  environment. 

LEARNING: 

Be  able  to  learn  and  apply  new  models,  techniques,  and 
technologies  as  they  emerge,  and  appreciate  the  necessity 
of  such  continuing  professional  development. 
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Outcomes  at  graduation 


TECHNOLOGY: 

Be  able  to  analyze  a  current  significant  software  technology, 
articulate  its  strengths  and  weaknesses,  compare  it  to 
alternative  technologies,  and  specify  and  promote 
improvements  or  extensions  to  that  technology. 
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Curriculum  architecture 
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1  GSwE2009  Release 


•  Version  1 .0  was  released  to  the  international  SwE 
community  Sept.  30,  2009. 

•  Delivered  to  US  DoD  OSD 

•  Delivered  to  ACM  EB,  IEEE  CS,  INCOSE,  and  CAT 

•  The  document  is  available  online  at 

www.gswe2009.org/curriclunn/recommendations/document.pdf 


|  Post-version  1.0  governance 


1 .  Understand  the  current  state  of  SWE  graduate 
education  (November  2007) 

2.  Create  GSwE2009  0.25  with  a  small  team,  suitable  for 
limited  review  (February  2008) 

3.  Publicize  effort  through  conferences,  papers,  website, 
etc  (continuous) 

4.  Create  GSwE2009  0.50  suitable  for  broad  community 
review  and  early  adoption  (October  2008) 

5.  Create  GSwE2009  1 .0  suitable  for  broad  adoption 
(2009) 

6.  Transition  stewardship  to  professional  societies  (2009) 

7.  Foster  adoption  world-wide  (2009  and  beyond) 
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Implementation  help 


•  Comparison  of  existing  graduate  software  engineering 
programs  with  GSwE2009  recommendations  -  know 
how  big  the  gap  is  between  recommendations  and 
practice 

•  Strategies  recommended  by  the  authors  to  implement 
GSwE2009 

•  Hypothetical  modifications  of  existing  programs  to  more 
fully  satisfy  GSwE2009 

•  Workshops  targeted  to  department  heads  and  faculty 

•  Build  a  business  case  for  GSwE2009 

•  Facilitate  curriculum  modification  or  development  to  22 
alian  with  GSwE2009  recommendations 


9  Preparing  for  Post  1.0  World 


•  GSwE2009  vl  .0 — primary  recommendations  document — 
released  Sept  30,  2009 

•  Two  companion  documents  for  GSwE2009  will  be 
delivered  October  2009: 

•  Implementation  guidance  organized  as  an  FAQ 

•  Comparison  of  current  SwE  programs  to  GSwE2009 

•  Primary  recommendations  are  typical  of  what  professional 
societies  traditionally  shepherd 

•  Implementation  guidance  and  comparisons  are  less 
typical  of  what  professional  societies  traditionally 
shepherd 
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Possible  long-term  governance 


•  ACM,  IEEE  CS,  INCOSE,  NDIA  SE,  and  the  Brazilian  Computer 
Society  all  participating  in  GSwE2009  creation. 

•  Joint  ACM  and  IEEE  CS  (primary),  and  INCOSE  (supportive) 
governance  model  for  Curriculum  Recommendations  is  desirable 
with  periodic  updates. 

•  Small  volunteer  body  to  provide  periodic  updates  of  FAQ  and 
comparisons  materials  with  website  support  including  forums, 
wikis,  and  other  open  collaboration  structure. 

•  Implementation  workshops  at  conferences,  summer  faculty 
workshops,  and  other  activities  would  promote  adoption.  The  CAT 
is  currently  seeking  assistance  from  the  NSF  to  support  these 
workshops. 
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•  INCOSE  sponsored  a  graduate  systems  engineering  (SE) 
reference  curriculum  published  in  2007. 


•  The  SE  curriculum  development  process  did  not  have  the  scale  of 
participation  that  GSwE2009  has  and  is  limited  by  the  fact  that  the 
INCOSE  SE  Body  of  Knowledge  (see  http://g2sebok.incose.org)  is 
much  less  robust  and  mature  than  SWEBOK. 


•  INCOSE  would  like  to  mature  the  SE  body  of  knowledge,  which 
would  be  a  strong  foundation  on  which  to  base  an  upgraded  SE 
curriculum. 


•  The  U.S.  Department  of  Defense  is  considering  sponsoring  a 
project  to  update  and  mature  the  SE  body  of  knowledge  with 
INCOSE  and  create  a  mature  SE  reference  curriculum.  The  effort 
would  be  similar  to  GSwE2009  with  open  collaborative  international 
participation  and  fully  shared  resulting  intellectual  property. 


•  Other  professional  societies  would  be  welcome  to  participate. 
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Summary 


•  GSwE2009  vl  .0  delivered  September  30,  2009. 

•  Professional  societies  are  considering  taking  ownership 
of  the  curriculum  after  it  is  published. 

•  IEEE  CS  Educational  Advisory  Board  voted  to 
become  sponsors  of  GSwE2009  October  2009 

•  GSwE2009  companion  documents  scheduled  for 
release  Fall  2009 

•  Adoption  workshops  anticipated  summer  201 0 
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MEMORANDUM  FOR  CHAIRMAN  OF  THE  JOINT  CHIEFS  OF  STAFF 
UNDER  SECRETARIES  OF  DEFENSE 
GENERAL  COUNSEL  OF  THE  DEPARTMENT  OF 
DEFENSE 

DIRECTOR,  ADMINISTRATION  AND  MANAGEMENT 
DIRECTOR.  PROGRAM  ANALYSIS  AND  EVALUATION 
DIRECTORS  OF  THE  DEFENSE  AGENCIES 
CHIEF  INFORMATION  OFFICER,  U  S.  STRATEGIC 
COMMAND 

CHIEF  INFORMATION  OFFICER,  U  S.  JOINT  FORCES 
COMMAND 

DIRECTOR  OF  NATIONAL  INTELLIGENCE  CHIEF 
INFORMATION  OFFICER 

CHIEF  INFORMATION  OFFICERS  OF  THE  MILITARY 
DEPARTMENTS 

SUBJECT:  DoD  CIO  Governance  Board  Restructure  Implementation 

With  the  support  of  your  staffs,  this  office  has  completed  an  effort  to  streamline  the 
DoD  Information  Management/Information  Technology  governance  structure  to  reduce 
duplicative  or  unneeded  governing  bodies,  and  to  leverage  tiered  accountability  and 
delegate  authority  wherever  possible.  The  attached  document  provides  this  new  structure, 
composed  of  forums  within  the  authorities  of  the  DoD  Chief  Information  Officer  (CIO)  via 
DoD  Directive  5 144.01 .  This  new  structure  provides  for  better  oversight  of  the 
development,  review  and  endorsement  of  initiatives;  empowerment  of  lower  forums;  and 
strengthening  of  the  decision-making  and  advisory  processes  between  and  among  the  DoD 
CIO  and  the  DoD  CIO  community. 

Implementation  of  this  new  governance  structure  is  effective  immediately.  Points  of 
contact  arc  Ellen  Law  (cllcn.lawfa  osd.mil:  703-699-0118)  and  Ron  Alsbrooks 
(Ronald.alsbrooks.ctr@osd.mil:  703-699-0101). 


J>4vid  M.  Wennergren 
Performing  the  Duties  of  the 
ASD(NII)/  DoD  CIO 


Attachment: 

CIO  Governance  Board  Restructure 


Q 


FEDERAL  INFORMATION  SHARING  INIHAUVE 
INFORMATION  SHARING  STRATEGY/  IMPLEMENTATION 
PLAN 
CUI 


PROG  RAM  ANALYSIS  & 
SUPPORT 

CIO  GOVERNANCE 
INFORMATION  QUALITY 
DOMAIN  NAME  &  WEB 
POLICY 

PRIVACY  IMPACT 
ASSESSMENTS 
E-GOVERNMENT(OMB) 
RESOURCE  MANAGEMENT 
SECTION  508 
IT  WORKFORCE 
ACADEMIA  OUTREACH 
OMB  LIAISON 
GAO  LIAISON 
CONG  RESSIONAL  UAISON 
CIO  ROLE  -  8000.01 
FORMS  MG MT 
INFO  COLLECTION 
WEB  ADMIN/  MG  MT 


RECORDS  MG  MI- 
DATA  STRATEGY/  COIs 
SERVICES  STRATEGY/ 
SOA 

NCES,  MCEITS>  CANES 
IDENTITY  MG  MT.  & 
ATTRIBUTE  SERVICES 
SPEC  IAL  ASSTTO  THE 
VCJCS 

CLOUD  COMPUTING 
WEB 2.0-  DoDUse 
DODAPPS  STORE 


STRATEGIC  PLAN 
PERFORMANCE 
MEASUREMENT 
POUCY  FRAMEWORK 
FUTURE  OF  THE  CIO 
INTERNET  G  O  VERNANC  E 
AGENCY/ ASSOC. 
OUTREACH 
FEA  UAISON 


TITLE 40/  CCA  OVERSIGHT 
CPI/  LSS 

INDUSTRY  OUTREACH 
CLEARINGHOUSE 
ACQUISITION  SUPPORT 
C  OTS  TECH  REVIEW  & 
MARKET  RESEARCH 
INFORMATION  SUPPORT 
PLANS 

ENTERPRISE  SYSTEMS 
HEALTH  CARE,  ERPs, 

DIM  HRS,  etc. 
INTEROPERABILITY 


ENTERPRISE  SOFTWARE 
INITIATIVE 

TT  ASSET  MANAG  EMEISTT& 
COMMODITY  BUYING 
ITINVESTMENT 
MA  NAG  EM  ENT 
ENTERPRISE  G  UIDANC  E 
BOARD 

DoD  IT  INFRASTRUCTURE 
UBRARY(ITIL) 
COMPUTING 

INFRASTRUCTURE 

DITPR 

DBSMC  -  BTA/IRB 
UAIASON 
NGEN  OVERSIGHT 
jOINTBASING 
CPM  COORDINATION 


NETOPS 

IPv6 

UNIHED  C  APABIUT1ES 
ENTERPRISE  OPERATIONS  & 
OVERSIG  ITT  CO  MMITTEE 
OSD  CIO  UAISON 
PACC 

RISK  MANAGEMENT 
GIG  WAIVERS/ CONN  EC  HON 
APPROVAL  PROCESS 


DOD  FEDERATED  ENT.  ARCH 
STANDARDS  (DISR) 

DoD  INFO  ENT  ARCH  2.0 

DADAF2.0 

DARS 

REFERENCE  ARCH  SUPPORT 


JULY  2009 


CIO  Governance  Framework 


DoD  CIO 


I A  Architecture  approval  and 

engineering  guidance 

IA  Standards  and  configuration 

management 

Assure  integration  and 

compliance  of  capabilities  & 

solutions 


Architecture  approval  at  the 
enterprise  level 

Manage  architecture  federation 
Define  architecture  compliance 
criteria 

Maintain  authoritative  sources  of  IT 
standards  and  implementation-level 
specifications 


•  ES  strategic  planning,  policy  & 
i  implementation 

•  Data/Service  strategies  coord 

•  Solution  synchronization 

•  Information  sharing 

•  Servioe  registries 

•  Interoperability 

•  ES  to  tactical  edge 


-  Interoperability  and  supportability 


Direct  Reporting 
Complia  nc  e/Coo  inclination  ■ 


ASRG  Mission 


The  Architecture  and  Standards  Review  Group 
(ASRG)  serves  within  the  Department  of  Defense 
(DoD)  Chief  Information  Officer  (CIO)  Enterprise 
Governance  framework.  The  ASRG  is  subordinate  to 
the  DOD  CIO  EGB.  It  is  chartered  to:  review 
architecture  policy  and  guidance;  identify  DoD 
Information  technology  (IT)  technical  standards; 
oversee  IT  standards  management;  review 
architectures  and  enforce  architecture  policy;  oversee 
DoD  EA  Federation;  and  enforce  DoD  Information 
Enterprise  Architecture  (IEA)  compliance. 


ASRG  Organizational  Structure 


Ad  Hoc  j 
WGs 


ASRG 


i  Standards 
i  Principals 


Authoritative  Sources  of  Approved 
Arc  hHectuies&  Standards 


t  t  t  t 


ii  i 

pi  jj 

DoD  Policv 

DoDD  5000,  DoDD  4630,  DoDD  8210, 

CJCSI  6212,  CJCSI  3170 

https//  da  isl.a  imy.mil/  IER2/ 


a  @ 


p 


O': 


*  ..  -  \J  H 


fr  - 


1?  x 


1  *)  (  T  h  A  I 


1 


pSnt(P^l^uPAr  c^V'^UCrl  Vi 


Contact 


Help 


Member  Login 

User  ID  (Forgot?) 

jhttps :  //dars  1 .  ar  my .  m  i  I/1ER2/ 

Password  (Fo^ot?) 


Login 


CAC  Login 


No  Account?  Sign  Up 


News  &  Events 
8/ 18/2009 

DoDAF  2.0  Meta-model  (DM2) 
Walkthrough  (Details) 

9/1/2009  -  9/2/2009 

DARS  user  group  attendees,  please  see 
attached  lunch  order  form  (Details) 


9/ 1/2009 

DARS  User  Group  Agenda  (Details) 


Welcome 


m  ^rE 


sesee: 


DARS  provides  for  registration  and 
cataloging  of  DoD  Architectures 

DARS  Enables: 

•  Overview  and  Summary  (AV-1) 
registration,  creation  and  editing 

•  Discovery  metadata  creation  and 
search 

•  Alignment  with  federation  reference 
models 

•  Visualization  of  the  federated 
Enterprise  Architecture  (EA) 

•  Access-controlled  data  storage 

•  Architecture  data  management 


Quick  Links 


General  Public  Documents 
EA  Conference  Briefs 
Manage  Communities  &  Users 
Change  Request  Form 
Tutorials 


OLASSI  FI  ED/FOUO 


The  Department  of  Defense  DoD  Privacy  Policy 


https/  /  d  isronline  .d  isa  .mil/ 


©*©'B  ■  -dh 


Distort  line 


DoD  Information  Technology  Standards  Registry 

UNCLASSIFIED 


FAQs  — 
DISR  Calendar 
Reports  S  Archives 


DISRonline  Home  Contacts  Guidance  Links  Change  User  Info  Register  CAC  Create  RFC  Problems?  Need  Help?  Log  Off 


Standards  Mgmt 


Profiling 


DISRonline  will  be  offline  beginning  Thursday,  22  October,  through  the 
following  week. 

ITSC 


Classic  Search 


Change  Request 
Standards  Management 


Working  Groups 


Configuration 

Management 

□  ISA  Home 


Information  Technology  Standards  Committee  jpocsi 


•  Charter  (Signed) 

•  Standard  Operating  Procedures  fSQFl  -  ITSC/ISC/TWG  2009-07-28 

ITSC  Briefing  Templates 

•  TWG  Briefing  Template  f2QQ8-Q2-2C0 

ITSC  Meeting  2009  10  21 

•  Agenda  /  Minutes 

o  Agenda 

o  ITSC  Open  Action  Items  f2QQ9-1Q-Q71 

•  Briefings 

o  Decision  Brief  -  DISR  and  DISRonline  Topics  fD  Bernardinit 
o  Information  Brief  -  "Data  Standards"  Management  for  the  DISR  -  fD  Brownl 

o  Information  Brief-  Update  on  M&S  Technical  Working  Group  -  fA  Henningeh 

o  Information  Brief-  IC/DoD  Enterprise  Standards  Governance  fJ  Makowkal 

o  Information  Brief-  ICESG  Pilot  09-3. □  fC  Guthrie.  M  Fettiti 

o  Information  Brief-  Architecture  &  Standards  Review  Group  fASRGl  Briefing  fA  Golombekl 

o  Information  Brief-  Net-Centric  Operations  Industry  Consortium  fNCQICi  Briefing  fD  Browni 


ITSC  Meeting  2009  00  24 

•  Agenda  /  Minutes 

o  Agenda 

o  ITSC  Open  Action  Items  f2QQ9-Q7-Q21 

o  Meeting  Minutes  QQQg-QE-SAi 

•  Briefings 

o  Decision  Brief  -  DISR  and  DISRonline  Topics  CD  Bernardinh 


U  NCLASSI  FI  ED/FOUO 


'Mfli 


GIG  CMB Governance  Stmcture 


Chair/Configuration  Manager:  DISA 


GTG  CM  Board 


Co-Chair(s): 


OUSD(AT&L),  OASD(NII), 
OASD(DCIO),  JS(J6) 


ASD(NII)/DCIO  USD  (AT&L)  USA  USN  USAF  USMC  USCG  Joint  Staff/J6  DISA 

NSA  DOT&E  DLA  DTRA  M&SCO  MDA  NRO  DARPA  STRATCOM  DIA 
NGA  JFCOM  DNI  CIO  J2  J4 


Management  Direction 

Priority  Setting 

Periodic  Review 

Adoption 

GESP 

Issue  Adjudication 

Project  Oversight 

Coordination 

ASRG/ITSC/DISR 

GTG/GTG  Online 

Recommend 

Cross  Functional  Integration  Technical  Collaboration 

Integration 

Promulgation 

ASD(NII)/DNI/ICSR 

Services 

Agencies 

Stakeholders 

Wiki  Comment 
&  Review  Cycle 


Non-Attribution 


Open  Source 
Project  Model 


EWSE  IPTs 

NetOps 

■ 1 

k/\N 

Annual  GIG 
Technical 
Solution  Set 


Development 


Data  / 
Services 


GESP  GESP  GESP  GESP 


Computing 

Infrastructure 


Groups 


GESP  GESP  GESP  GESP 


Secured 

Availability 


Comm. 


GESP  GESP  GESP  GESP 


http//  www.disa  .mil/  ge/  ge3/  ge33.html 


ABOUT  DISA  OUR  ORGANIZATION  STRUCTURE  GIG  GE 


MISSION 


FOR  MORE  INFORMATION 

(703)  681-2596,  DSN  761 


GE33  -  INTEROPERABILITY  STANDARDS  DIVISION 


BOOKMARK  ^  t ... 


+  About  DISA 
Our  Work 
Our  Leaders 

Our  Organization  Structure 
Our  Strategy 
Our  History 

Serving  Our  Community 
Policy/Publication  Information 
Legal  and  Regulatory 


OFFICE  LINKS 


+■  GE1  -  Business  Management  Division 
^  GE3  -  Systems  Engineering  Center 
+-  GE31  -  End-to-End  Architecture 
Division 

+■  GE32  -  Horizontal  Engineering 
Division 

+  GE33  -  Interoperability  Standards 
GE331  -  Standards  Engineering 
Branch 

GE332  -  Standards  Management 
Branch 

GE333  -  Net-Centricity, 
Requirements,  Analysis  & 
Assessments  Branch 

-fr  CiFAd.  _  Mnrl^linn  snrl  ^imi  ilsrf-inn 


Identify,  aid  development  of,  promulgate,  and  catalog  information  technology 
standards  and  standards  profiles  to  meet  DISA  product  and  service  priorities  and 
the  needs  of  DoD,  with  emphasis  on  commercial  and  open  systems  standards. 

FUNCTIONS 

•  Lead,  coordinate,  and  influence  information  technology  (IT)  standards  engineering  efforts  to  address  DOD  and  DISA  requirements. 

•  Prioritize  IT  standards  efforts  and  reallocate  resources  as  necessary  to  support  DISA  prioritization. 

•  Coordinate  DoD  efforts  to  pursue  open  systems  standards  that  meet  DoD's  greatest  needs  as  reflected  in  DISA  products  and  services. 

•  Maintain  the  DOD  IT  Standards  Registry  (DISR)  and  configuration  manage  the  lifecycle  of  the  standards  cited  within. 

•  Maintain  the  GIG  Technical  Guidance  (GTG)  database  containing  GIG  Enterprise  Service  Profiles  that  detail  the  implementation  of 
enterprise  wide  systems  engineering  guidance. 

•  Evaluate  and  advise  on  implementation  of  IT  standards  and  key  interface  parameters. 

•  Identify  and  arbitrate  resolution  of  interoperability  and  standards  issues  within  DISA  and,  as  resourced,  DoD. 

•  Support  the  Joint  Staff  and  Assistant  Secretary  Defense  (National  Information  Infrastructure)  in  the  assessment  of  all  Information 


DODAFV2.0  Focus 


Results:  Better 
ANALYSIS  and 
Decisions. 


Focus: 

architecture  DATA 
not  Products 


RefKencc  Models  FuH*^ 


DoDAF  V2.0  provides  overarching  architecture  concepts,  guidance,  best 


presentation  Techniques 


All  Viewpoint 

Overarching  aspects  of  architecture  context  that  relate  to  all 


DoDAF  V2.0  Viewpoints  That  fit-the  Purpose 


^  Capability  Viewpoint 

Articulate  the  capability  requirement, 
delivery  timing,  and  deployed  capability 


Operational  Viewpoint 

Articulate  operational  scenarios, 
processes,  activities  &  requirements 


Services  Viewpoint 

Articulate  the  performers,  activities, 
services,  and  their  exchanges  providing 
for,  or  supporting,  DoD  functions 

Systems  Viewpoint 

Articulate  the  legacy  systems  or 
independent  systems,  their  composition, 
interconnectivity,  and  context  providing  for, 
or  supporting,  DoD  functions 


<D  o. 

?§ 

3  £■ 

CD  © 

>w 


£2  o 

3 

*  i  s 

3  =  <D 
<A  3 
fi>  2. 

S  o 

3 
(A 


O  O'  CD 

I  £< 


E.  £ 

0)  CD 
=;  CD 

3  O  _ 
0)  ^ 
c/)  "o  3 

<  D)  *2. 
(/)  O"  CD 

^  “  o 


~o 

(/> 


(D 

CD  ^ 
?  $ 
s  $ 

®  =  "§ 
3  <  W)  CD  is. 

3?3£££ 
o 


CD  fi> 
(A  CQ 
(A  CD 


(fi 


O 
3 
0) 

3  “ 
3  "5.  s 

CD  CD  a 

3.  3  o 

O!  CD  D) 
“  3  "U 

^  !-►  ft) 

s&l 


Architecture  viewpoints  are  composed  of  data  that  has  been 
organized  to  facilitate  understanding. 
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Defense  Information  Enterprise  Architecture  Release 


The  Defense  Information  Enterprise  Architecture  version  1.0  (DIEA  1.0)  provides  a  common 
Defense  Information  Enterprise  foundation  to  support  accelerated  Department  of  Defense  (DoD) 
transformation  to  net-centric  operations.  It  presents  the  vision  of  net-centric  operations  and 
establishes  near  term  priorities  to  address  critical  barriers  that  must  be  overcome  in  order  to 
achieve  the  vision. 

The  Defense  Information  Enterprise  Architecture  consolidates  underlying  DoD  Net-Centric  policies 
to  provide  guidance  for  all  DoD,  across  all  portfolios,  enabling  informed  discussions  among 
decision-makers  about  key  issues,  and  underpinning  process  improvements  throughout  the 
Department.  Defense  Information  Enterprise  Architecture  1.0  highlights  the  key  principles,  rules, 
constraints  and  best  practices  to  which  applicable  DoD  programs,  regardless  of  Component  or 
portfolio,  must  adhere  in  order  to  enable  agile,  collaborative  net-centric  operations. 
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This  website  represents  the  main  method  for  distributing  Defense  Information  Enterprise 
Architecture  1.0.  The  full  set  of  Defense  Information  Enterprise  Architecture  1.0  products  are 
available  from  the  left  side  menu  entitled  "DIEA  1.0  Products"  where  users  can  access: 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Systems  Engineering  Needs  of  the 
DoD  Architecture  Framework 

Report  of  the 

Architecture  Frameworks  Working  Group 
Systems  Engineering  Division 
National  Defense  Industrial  Association 

Co-Leads 

Carl  Siel,  ASN-RDA  CHSENG 
Joe  Kuncel,  Northrop  Grumman 
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AFWG  Purpose  and  Products 


Purpose 

•  Recommend  changes  and  additions  to  the  DOD 
Architecture  Framework  (DoDAF)  and  related 
standards  that  will  improve  support  for  DOD  systems 
engineering,  development,  and  acquisition. 


Products 


Final  report  and  briefing  of 

•  Analysis  of  DoDAF  satisfaction  of  SE  needs  ^ dodaf »2.0 

•  Conclusions 

•  Recommendations  for  improvement 
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AFWG  Members 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Name 

Organization 

Carl  Siel  (Co-Leader*) 

US  Navy  Chief  Systems  Engineer,  ASN-RDA-CHSENG 

Joe  Kuncel  (Co-Leader) 

Northrop  Grumman 

Ajit  Narayan 

Northrop  Grumman 

Chris  Phelps 

Sumaria 

Cliff  Whitcomb 

US  Naval  Post  Graduate  School 

David  Putman 

BAE  Systems 

Diane  Hanf 

Mitre  Corp. 

Elizabeth  Luvender 

Mitre  Corp. 

Hal  Wilson 

Northrop  Grumman 

Jennifer  Rainey 

Johns  Hopkins  University-Advanced  Physics  Laboratory 

John  Palmer 

Boeing 

Kristin  Giammarco 

US  Army  AMC  &  US  Naval  Post  Graduate  School 

Robert  Curry 

Raytheon 

Scott  Osborne 

Savvee  Consulting,  Inc. 

Thomas  Murphy 

Silver  Bullet  Solutions,  Inc. 

*  AFWG  founder  and  sponsor 
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SE  Needs  of  DoDAF 


•  SE  Needs 

•  Standard  architecture  modeling  methodology 

for  greater  reuse/sharing,  more  efficient/standardized  modeling 

•  Greater  definition  and  standardization  of  architecture  elements 

incorporation  of  Services’  “architecture  elements  lists” 

•  Executable/simulatable  architecture  models 

for  early  and  inexpensive  architecture  verification  and  validation 

•  Composable/decomposable  architectures 

for  multiple  levels  of  abstraction  for  hierarchy  of  stakeholders 

•  More  reusable  architecture  models 

faster,  more  efficient,  more  standard  architecture  development 

•  Standard  architecture  alternatives  analysis  method 

for  continuous  architecture  improvement 

•  Standard  architecture  modeling  notation  and  symbology 

better  architecture  comprehension  and  communication 

•  Auto-generation  of  systems  engineering  artifacts 

lowering  costs  by  leveraging  architecture  model’s  “authoritative  data” 
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Recommendations 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Standard  architecture  modeling  methodology 

•  Object-Oriented,  UML,  SysML,  UPDM  implementation  of  DM2 


Model 


Standardized 

1 

Architecture  Models 

rmodeled  by 


Meta-models 


UPDM 

^tension  of  UML/SyaML 


includes 


modeled  by 


Meta-meta-model 


Object-Oriented  p£ir=idiy/n  & 
Meta-Object  Facility 


A 


DoDAF  72 

UML  *  SJyaML 

Meta-Model 

Modeling  Language 

J 


Greater  Model.. 

•  Reuse  &  Sharing 

•  Communication 

•  Comprehension 

•  Precision 

•  Mature  Industry  Standards 

•  Broadest  Tool  Support 

•  Widest  Skills  Base 

•  Most  Academic  Support 

•  Most  Intuitive 

•  Greatest  Model... 

•  Scalability 

•  Flexibility 

•  Adaptability 
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Recommendations 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Greater  definition,  standardization  of  architecture  elements 

•  Integrate  JFCOM’s  standard  architecture  elements  into  the  DM2 

-  by  adding  elements  to  meta-model  OR 

-  by  enabling  user  extension  (profile)  of  meta-model 


Existing  DM2 
Element 

Recommended 

Element 


Conduct  Operational  Maneuver  and  Force  Positioning  Activity 


Meta-Model: 


IndividualType 

<1 - 

IndividualType  Extended 

Profile-Extended 
Meta-Model  Example: 


'ii  n\  \  \ 

«instanceO/»y  «inptar\ceQf»\ 
«instanc^Of»  /  «instanpeQf» 
«instanceOfy>  \  «jnstanceOf» 

//  /  «mstanceOf»  « ^stance Of » 


TaggedValue 

name:  string 
-  value:  string 

-t - f - 1- 


— \ — V 


System 

TOW/Anti^Tan^Systerr^  '  ^  ^ 

- L - f - h-?- - t — \ - \ - 4 


Automate^ Digital  Netwof^c  S^stei\i  \ 


\  \  \  \ 

icgtiorK  System  \ 

— t - 1 - 


Existing  DM2 
Element 

Recommended 

Element 

Profile  Extension 
Element 


Global  Combat  Support  S’ 


\ 

\ \ 


\  \ 
ia 


\  \ 


Assess  toHjhURm^ffects  on  Ops  Targets  Activity 

<3- 

Provide  CommanderXjnpqt  to  Initial  Courses  of  Action  Activity 

Activity 

\  \  \ 

Assess  Munitions  Effects  on  Ops  Targets  Activity  \ 

1  \  \ 

Provide  Commander's  Input  to  Initial  Courses  of  A^ion  Achjvity 


Conduct  Operational  Maneuver  and  Force  Positioning  Actjvity 


Protect  Personnel  From  Biological  Contamination  Effects  Activity 

Protect  Personnel  From  Biological  Contamination  Effects  Activity 
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Recommendations 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Executable/simulatable  architecture  models 

•  Add  behavioral  semantics  for  state-machine  ... 

Meta-Model: 


Performer 

m  whole-part 

State  Machine 

whole-part 


\  whole-part 
1  .  \  1-* 


Existing  DM2 
Element 


Recommended 

Element 


State 

uiaio  i  iai  uuhiwvci  i  cup 

l 

T  ransitionStatejOverlap 

Transition 

Event 

wholt 

3 -part 

EnterStateActivi^ 
\  Do^t^teActjvityd^rlap 


IvOverlap./'^  A 
. _  1  .  whol 


T 


whole-part 


_ M  / _ 

l 

Activ  ity 

i 

«instanceOf» 

1 

- - 7 

Condition^ 
_ L _ * 

- t - 

3 

/ 

/ 

/ 

- 7 - 

Model  Example: 


«instanceOf» 
/ 


«instanceOf» 
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Recommendations 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Executable/simulatable  architecture  models  (cont’d) 

•  And  add  behavioral  semantics  for  activity  definition  ... 


\  /  i  /  / 

«instanceOf»  /  I  «instanceOf»  / 

Model  Example:  |  /  '  / 
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Recommendations 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Composable/decomposable  architectures 

•  Add  abstraction  relationship... 


Meta-Model: 


Model  Example: 


Note:  DM2  deemed  to  satisfy  need  for  structural 
composition/decomposition  of  architectures 
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Recommendations 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


More  reusable  architecture  models 

•  Extend  physical  exchange  standard  (PES)  to  include  diagrams  exchange  (XMI  for 
UPDM  already  includes  diagram  exchange) 

•  Standardize  DARS  on  PES  (prefer  XMI) 

•  Add  patterns  and  frameworks  support  to  DARS 


Domain  Modeling 


Model  Standardization 


Model  Reuse/Sharing 


shared 

to/from 


-  federated 

-  standardized 

-  includes  architecture 

fragments 

frameworks 

patterns 


Format  UML/SysML/UPDM 

<or> 

Tool  Specific  Notation 


UML/SysML/UPDM  XMI 
<or> 

DoDAF  PES  XML  with 
diagram  extensions 


UML/SysML/UPDM  XMI  Files 
<or> 


DoDAF  PES  XML  Files 
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Recommendations 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Standard  architecture  alternatives  analysis  method 

•  Extend  meta-model  with  parametric  analysis  semantics 

•  Standardize  on  S El’s  Systems  and  Software  ATAM 


Meta-Model: 


Performance  Measure 


Maintainability  Measure 


Adaptability  Measure 


etc... 


Model  Example: 


Backup  Forward 
Surveillance  System 


Forward  Surveillance 
System 


Existing  DM2 
Element 


Physical  Measure 


Recommended 

Element 

Measure  Type 

Measure 

-  units:  string 

<r 

«instanceOf» 

-  value:  string  < 

/ 

overlapjype 

- 7^ - 


«instanceOf»  ^ 

/  «instanceOf» 


« instance  Of» 
\ 


Mobile  Surveillance 
System  M7BF 

value:  int  =  3000 


1+MobilleMT, 


Computed  Measure 

] - 

-  computation:  string 

^  A 

-> 


+Stc 


Computation  parsed  and  calculated  by  modeling  tool 


tionaryMTBF 


Stationary  Surveillance 
System  MTBF 

value:  int  =  2000 


«instanceOf» 

/ 

/ 


Backup  Base 
Surveillance  System 


Base  Surveillance 
System 


SystemMTBF 

value  =  1800 

computation  =  “value  =  1/((1/MobileMBTF. value  *  (1  +  1/2))  + 
(StationaryMTBF. value  *  (1  +  1/2)))” 
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Recommendations 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Standard  architecture  modeling  notation  and  symbology 

•  Establish  UML/SysML/UPDM  as  standard  notation  ... 
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Recommendations 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Standard  architecture  modeling  notation  and  symbology 

•  Extend  meta-model  with  symbolic  representation  semantics  .... 


Meta-Model: 


etc 
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Recommendations 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Standard  architecture  modeling  notation  and  symbology 

•  Establish  DoD  Metadata  Registry-like  standard  symbology  library 

Model  Example  (notation): 


«block» 

LF  Comm  System 

«block» 

GIG  System 

«block» 

SBN  System 
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Recommendations 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Auto-generation  of  systems  engineering  artifacts 

•  Extend  meta-model  with  user-definable  extensions  (tagging) ... 


Meta-Model: 


«instanceOf»  «instariceOf» 


Existing  DM2 
Element 

Recommended 

Element 

User  Extension 
Element 


User  Extended  Meta-Model  Example: 


Model  Example: 
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Recommendations 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Auto-generation  of  systems  engineering  artifacts  (cont’d) 

•  Establish  standard  model  reporting  capability 

Artifacts 


Detailed  Architecture 
Model  with  Custom 
Meta-model  Extension 
Information 


Standard 


Standard  Reporting 
&  Scripting  Tools 


Web-Based 
Model  Views 


Interface 

Specifications 


Architecture 

Descriptions 


Test  Procedures 

Acquisition  & 
Program  Plans 


Architecture 

Metrics 


Requirements 
Trace  Matrices 


Business  Process 
Automation  Scripts 
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Summary 

In  summary... 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


DoDAF  v2  improves  on  satisfaction  of  SE  needs,  but 
systems  engineers  need  greater  definition  and 
standardization  of  semantics  and  methods  that  are 
important  to  them. 


Comments,  questions,  feedback  are  solicited.  Contact... 
Joe.Kuncel@ngc.com,  402-682-4772 
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Human-Centered  Design 
in  Systems  Engineering: 
Human  View  Methodology 


Robert  J.  Smillie,  PhD ,  CPE 
Space  and  Naval  Warfare  Systems  Command 

NDIA  Systems  Engineering  Conference 
29  October  2009 


BS 


srm/AR 


Objective  and  Approach 


▼  Examine  dynamic  aspects  of  Human  View  as  an 
effective  methodology  for  Human  Systems  Integration 
(HSI)  practitioners  coordinating  and  collaborating  with 
systems  engineers. 

▼  Use  data  from  system  development  effort  to  build 
Human  Views. 

T  Use  modeling  and  simulation  to  analyze  dynamic 
operator  elements  of  the  system  to  augment  Human 
View  process. 


HSJ 


11/4/2009 


A 


2 


srm/AR 

'im 


In  Practice 


▼  Design,  development,  and  production  of  large  complex 
systems  requires  the  HSI  practitioner  to  ensure  that 
HSI  results,  e.g.,  the  task  analysis,  are  communicated 
in  a  language  that  the  systems  engineer  understands. 

T  An  architecture  framework  provides  that 
communication  medium. 


HSJ 
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Architecture  Frameworks 


srm/AR 

yw 


▼  Defines  common  approach  for  development, 
presentation,  and  integration  of  architecture  descriptions. 

▼Architecture  frameworks  are  used  by  systems  engineers 
to  provide  a  common  set  of  products  and  product 
descriptions  for  representing  systems. 

▼Current  frameworks  fail  to  capture  the  human-centered 
design  aspects  needed  to  ensure  the  effectiveness  of 
human  operated  systems,  such  as  users  requirements, 
capabilities  and  limitations. 


SPAWN* 


Department  of  Defense  Architecture 
Framework  (DODAF) 


▼  DoDAF  defines  different  views  that  breakdown  a  complex 
system  into  specific  categories: 

■  All  View  -  Describes  the  Scope  and  Context  (Vocabulary)  of  the 
Architecture 

■  Operational  View  -  Identifies  What  Needs  to  be  Accomplished 

■  Systems  and  Services  View  -  Relates  Systems,  Services,  and 
Characteristics  to  Operational  Needs 

■  Technical  Views  -  Prescribes  Standards  and  Conventions 

▼  Each  of  the  four  views  depicts  certain  architecture 
attributes  -some  attributes  bridge  two  views  and  provide 
integrity,  coherence,  and  consistency  to  architecture 
descriptions. 

▼  However,  none  of  these  conventions  focus  explicitly  on  the 
human  element  -  by  adding  a  Human  View  to  the 
architecture  framework,  an  understanding  of  the  human  role 
in  systems/enterprise  architectures  is  included. 


11/4/2009 


A 
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SPAWAR 

$ 

*  Emergence  of  the  Human  View 

T  Early  efforts  to  represent  humans  in  architecture 
products  focused  on  human  role  and  activities. 

■  Hildebrand  and  Adams,  2002 

■  Handley,  2006 

T  Additional  analytical  efforts  in  both  Canada  (DNDAF)  and 
United  Kingdom  (MoDAF)  have  been  concerned  with  how 
to  include  human  activities  in  architecture  framework. 

■  Baker  et  al,  2006 

■  Bruseberg,  2008 

T  Human  View  methodology  provides  HSI  practitioner  a 
mechanism  to  convey  an  understanding  of  human  role  in 
systems/enterprise  architectures  to  systems  engineers. 

HSI 
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SPAWAR 

Human  View 


▼  Purpose  is  to  organize  information  into  a  framework  about 
how  the  human  functions  in  the  system  in  order  to  model  the 
impacts  of  human  performance  from  tasks,  personnel,  and 
system  resources. 

▼  Provides  a  set  of  products  which  captures  information  on 
Capabilities,  Constraints,  Tasks,  Roles,  Networks,  Training, 
and  Metrics,  which  are  integrated  with  a  dynamic  model  used 
to  determine  human  risk. 

▼  By  using  the  Human  View 

■  It  ensures  that  the  human  is  fully  considered  in  the 
architecture  by  structurally  incorporating  them  into 
engineering  planning. 

■  It  provides  human-system  parameters  that  can  be  used  to 

minimize  human  risk  with  the  overall  system.  H£j 
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SPAWN* 


Human  View  Product  Descriptions 


▼  HV-A  :  Concept  -  A  conceptual,  high-level  representation 
of  the  human  component  of  the  enterprise  architecture 
framework. 

▼  HV-B :  Contraints  -  Sets  of  characteristics  that  are  used 
to  adjust  the  expected  roles  and  tasks  based  on  the 
capabilities  and  limitations  of  the  human  in  the  system. 

▼  HU-C:  Tasks  -  Descriptions  of  the  human-specific 
activities  in  the  system. 

▼  HV-D:  Roles  -  Descriptions  of  the  roles  that  have  been 
defined  for  the  humans  interacting  with  the  system. 

▼  HV-E:  Human  Network  -  The  human  to  human 
communication  patterns  that  occur  as  a  result  of  ad  hoc 
or  deliberate  team  formation,  especially  teams 
distributed  across  space  and  time. 

▼  HV-F:  Training  -  A  detailed  accounting  of  how  training 
requirements,  strategy,  and  implementation  will  impact 
the  human. 

▼  HU-G:  Metrics  -  A  repository  for  human-related  values, 
priorities  and  performance  criteria,  and  maps  human 
factors  metrics  to  any  other  Human  View  elements. 

▼  HV-H:  Human  Dynamics  -  Dynamic  aspects  of  human 
system  components  defined  in  other  views. 


HV-A 

Concept 


HV-B 

Constraints 


participates  in 


HV-E 

Human 

Network 


contributes  to 


evaluates 


HV-G 

Metrics 


HV-H 

Human 

Dynamics 


' assesses 


HSJ 


11/4/2009 


8 


srm/AR 


Example:  HV-A 


▼  HV-A  is  a  conceptual,  high-level 

representation  of  the  human  component  of 
the  enterprise  architecture  framework.  Its 
purpose  is  to  visualize  and  facilitate 
understanding  of  the  human  dimension  in 
relation  to  operational  demands  and  system 
components. 


Pictorial  depictions  of 
the  system  and  its 
human  component. 

High  level  indicators  of 
where  human 
system  interactions 
may  occur. 


Textual  descriptions  of 
the  overall  human 
component  of  the 
system. 


Team  Interaction 
Representations 


OV-1 

Operational 

Concept 


[Architecture 

Representation 


HV-A 

Concept 


Role 

Representations 


Use  cases  which 

describe  the  humar 
process. 


HV-E 

Human 

Network 


...Shared  Situation 
Awareness, 

Basis  for  Commander’s 
Decisions 


briefing  team  " 


Commander’s  Daily 
Update  Brief... 

Collaboration  at  Sea 


information 
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SPAWAR 


Example:  HV-B 


▼  Manpower  Projections  (HV-B1) 
illustrates  predicted  manpower 
requirements  for  supporting  present 
and  future  projects  that  contribute 
to  larger  capabilities. 

Manpower  forecasting  to  allow  initial 
adjustments  in  training,  recruiting, 
professional  development, 
assignment  and  personnel 
management. 

Impacts  (and  timeframe)  related  to 
numbers  of  personnel,  personnel 
mix,  Military  Occupational 
Structure  Identification  (MOSIDs), 

Rank/level  distribution,  and, 
postings/relocations  of  personnel; 

Number  of  personnel  with  necessary 
Knowledge,  Skills,  and  Abilities 
(KSAs)  ‘ready  and  able’  to  supporl 
fielding  of  future  program. 


CAPABILITY  PACKAGE  A 


o 


o 


o 


o 


o 


o 


o 


PROGRAM  PORTFOLIO  B 


o  o 


o 


2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020 


This  diagram  maps  current  and  planned  projects  to 
capabilities.  For  each  year,  the  Initial  Operating 
Capability  (IOC)  and  Final  Operating  Capability  (FOC)  are 
indicated.  Manpower  requirements  for  each  year  can 
also  be  indicated  by  job  and  rank. 


PROJECT  2. 

w 

w 

PROJECT  3 

# 

o 

o 

o 

o 

HV-B  (Personnel) 


HV-B(HFE) 


11  2012  2013  2014  2015  2016  2017  2018  2019  2020 


▲  A 

L  a 

Personnel 

Role  Requirements  &  j 

Constraints 

Personnel  Limitations  1 

▼  ^ 

r 
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Example:  HV-E 


▼  The  HV-E  captures  the  human  to 
human  communication  patterns 
that  occur  as  a  result  of  ad  hoc  or 
deliberate  team  formation, 
especially  teams  distributed  across 
space  and  time. 

Role  groupings  or  teams  formed, 
including  the  physical  proximity 
of  the  roles  and  virtual  roles 
included  for  specific  team  tasks. 

Type  of  interaction  -  i.e.,  collaborate, 
coordinate,  supervise,  etc. 

Team  cohesiveness  indicators  -  i.e., 
trust,  sharing,  etc. 

Team  performance  impacts  ■  i.e., 
synchronization  (battle  rhythm), 
level  of  engagement  (command 
directed). 

Team  dependencies  -  i.e., 

frequency/degree  of  interaction 
between  roles. 


HV-C 

Tasks 


HV-A 

Concept 
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Teams  & 
Interactions 


Information  Requirements 


OV-3 

Operational  Information 
Exchange  Matrix 


HV-E 

Human 

Network 


ns/ 


Technology  Options 
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RBK/  ASHORE 


Communication  and 
In  f  o  r  mat  ion  Syst  ems  Center 


Information  Manager 
Plan  Developer 


FWD/  AFLOAT 


COA  Planner  Lead 


\ 


Future  PI  ans  Center 


COA  Planner  Surface 


Intel  I  ig  en  c 


Maritime 

Developr 


This  diagram  indicates  the  interactions  across  teams  involved 
in  the  Commander’s  Update  Brief  process.  It  also 
specifies  the  communication  types  and  team  locations. 


SPAWAR 

y  Human  View  Interaction  with  DoDAF 
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SPAWN* 


Human  View  Case  Study 


T  Used  Improved  Performance  Research  Integration  Tool 
(IMPRINT). 

■  Stochastic  task-network  modelling  tool  to  help  assess  interaction 
of  people  and  system  performance  from  concept  and  design 
through  field-testing  and  system  upgrades.  (Mitchell  et  al,  2008) 

■  Helps  researchers  and  designers  evaluate  operator  mental 
workload  while  testing  alternate  system-operator  function 
allocations.  (Wickens,  1991) 

T  Purpose  of  the  dynamic  Human  View  is  to  capture  the 
interaction  of  the  human  system  components. 

■  An  effective  modeling  and  simulation  tool  can  assess  the  static 
Human  View  data  under  dynamic  situations  and  provide  the 
system  engineer  designers  with  a  robust  set  of  HSI  criteria. 
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SPAWAR 

V'"  Dynamic  Model  Elements 


Human  View  Product 

Data  Required  by  Simulation  Model 

HV-A  Concept 

Hypothesis  to  be  tested  by  the  model. 

HV-B  Constraints 

Selection  of  the  Moderator  settings  of 
Personnel  and  Stressors. 

HV-C  Tasks 

Generation  of  the  Network  Diagram 
composed  of  Tasks  and  Subtasks; 
Assignment  of  System  Interfaces  to  Tasks. 

HV-D  Roles 

Creation  of  Operator  list;  Assignment  of 
Operators  to  Tasks. 

HV-E  Human  Network 

Identification  of  Team  Functions  and 

Operator  Teams. 

HV-F  Training 

Selection  of  the  Moderator  setting  of 

Training. 

HV-G  Metrics 

Identification  of  Mission  Level  Time  & 

Accuracy  criterion  and  selection  of  Task 

Level  Time  &  Accuracy  standards. 

A 
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spawn* 


Method 


▼  Used  U.S.  Army’s  Future  Combat  System. 

▼  Created  experimental  model  in  IMPRINT. 

■  Operators  were  defined  by  the  Human  View  roles. 

■  Task  descriptions  were  used  to  create  a  network  diagram  for  a  specified 
mission. 

■  Task-role  combination  provided  the  operator  assignments. 

■  Performance  standards/measures  were  used  to  define  the  expected  task 
times,  accuracy,  and  outcomes. 

■  Constraints  determine  moderators  that  impact  performance  (e.g.,  heat, 
etc.). 

■  IMPRINT  outputs  provide  data  that  describe  overall  success/failure  of  the 
mission,  task  performance  completion  and  potential  errors,  and  operator 
workload. 

T  Results  used  to  support  systems  engineering  process  to 

ensure  human/operator  requirements  are  met. 


HSJ 
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SPAWN* 


Dynamic  Model  Inputs 


Interfaces  (HV-E) 


HV-C2  System  Interfaces 

Role 

System 

Platoon  Leader 

1  LCU  Centralized  Controller;  2  Centralized  Controller, 
Tactical  (UGS-T) 

Platoon  Sergeant 

Multifunction  Utility/Logistics  and  Equipment  Transport 
(MULE-T) 

Vehicle  Commander 

Squad  Leader 

3  Centralized  Controllers;  Small  Unmanned  Ground  Vehicle 
(SUGV) 

Robotic 

Armed  Rconnaissance  Vehicle  (ARV-A);  Class  1  unmanned 
Aircraft  System  (UAS) 

Team  Leader 

6  sets  Intelligent  Ground  Sensors  (UGS-U) 

Health  Care 

Driver 

Infantry  Carrier  Vehicle 

Infantry 

MK  44  30MM;  MK240  7.62MM; 

Tasks  (HV-C) 


HV-C  TASKS 

Squad  Level 

Conduct  Tactical 

Initiate  Road  March 

Move  Along  March  Route 

Maintain  March  Security 

Conduct  Scheduled  Halts 

Platoon  Arrives  at  Designated 

Operation 

React  to  Ambush 

Driver  Reacts  to  Ambush 

Ambush 

Vehicle  Commander  Reacts  to 
Ambush 

Infantry  Squad  Reacts  to  Ambush 

Platoon  Leader  Reacts  to 

Ambush 

fEromUBaFv'niUredPerSOnnel 

Disengage  From  An  Enemy 

Conduct  Resupply  Operations 

Conduct  Maintenance 
Operations 

Conduct  Consolidation  and 

Equipment  Vehi0'eSand 

|  IMPRINT  Pro  -  Analysis:  Tactical  Road  March  Version:  1 1 1  Mission:  Tactical  Road  March 

Roles  (HV-D) 


HV-D  Roles 

Abbreviation 

Role  Name 

MOS 

PL  02 

Platoon  Leader 

11 A 

PSG/VC  E7 

Platoon  Sergeant 

11B40 

VCE6 

Vehicle  Commander 

11B30 

SL  E6 

Squad  Leader 

11B30 

VCE5 

Vehicle  Commander 

11B20 

RBTIC  E5 

Robotic 

11B20 

TLE5 

Team  Leader 

11B20 

HCE4 

Health  Care 

68W10 

DVRE4 

Driver 

11B10 

INF  E4 

Infantry 

11  BIO 

CCSWE4 

Common  Close  Support  Weapon 

1 1  BIO 

A/GNR  E4 

Gunner 

1 1  BIO 

A-Tank  E4 

Anti  Tank 

1 1  BIO 

Metrics  (HV-G) 


Performance  Standards 
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SPAWAR 

$ 

Approach 

T  Baseline  simulation  was  executed  to  provide  expected 
levels  of  mission  performance  parameters  of  time  and 
accuracy. 

■  IMPRINT  provides  for  the  overall  development  of  a  network 
task  model  that  accounts  for  the  tasks  and  the  types  and 
numbers  of  operators  performing  those  tasks. 

■  It  also  provides  the  opportunity  to  examine  the  effects  of 
unexpected  outcomes. 

T  Simulation  was  run  multiple  times,  the  outcome 
measures  were  analyzed  in  terms  of  performance  and 
workload. 


BSJ 

A 
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SPAWAR 

V'"  Dynamic  Model  Outputs 


Operator  Workload 


Impact  of  Constraints 


Workload  Graph 


IMPRINT 


-  Platoon  Leader 

-  Platoon  Sergeant 
Vehicle  Commander 
Squad  Leader 

-Team  Leader 

-  Health  Care 

-  Driver _ 


Individual  Task  Performance 


Time 

Accuracy 

Overfill  M  |  &  1^^ 

Task 

Standard 

Minimum 

Maximum 

Mean 

Std.  Dev. 

%  Met 

Accuracy  Standard 

Accuracy  Measure 

%  Met 

Mission  Aborts 

%  Met  Both  Time  AND  Accuracy 

VI  psss  iron 

START 

00:08:00.00 

00:05:57.80 

00:07:29.40 

00:06:52.21 

00:00:25.76 

100.00 

90.00 

Percent  Steps  Correct 

100.00 

0 

100.00 

This  DOESThBfenHS  (J&rforrffanSfe  HlfSridn  of^0% 

Initiate  Road  March 

00:25:00.00 

00:23:01.37 

00:25:13.73 

00:24:08.51 

00:00:31.39 

96.00 

90.00 

Percent  Steps  Correct 

92.00 

0 

88.00 

This  does  NOT  meet  the  performance  criterion  of  90% 

Move  Along  March  Route 

00:20:00.00 

00:17:30.67 

00:20:07.57 

00:19:02.03 

00:00:33.38 

96.00 

90.00 

Percent  Steps  Correct 

100.00 

0 

96.00 

This  DOES  meet  the  performance  criterion  of  90% 

Report  Control  Measures 

00:20:00.00 

00:18:11.83 

00:20:22.81 

00:18:55.64 

00:00:34.64 

96.00 

90.00 

Percent  Steps  Correct 

92.00 

0 

88.00 

This  does  NOT  meet  the  performance  criterion  of  90% 

Maintain  March  Security 

00:20:00.00 

00:17:40.71 

00:19:59.11 

00:18:47.98 

00:00:34.83 

100.00 

90.00 

Percent  Steps  Correct 

100.00 

0 

100.00 

This  DOES  meet  the  performance  criterion  of  90% 

Conduct  Scheduled  Halts 

00:20:00.00 

00:18:00.54 

00:19:32.56 

00:18:56.32 

00:00:21.07 

100.00 

90.00 

Percent  Steps  Correct 

100.00 

0 

100.00 

This  DOES  meet  the  performance  criterion  of  90% 

Platoon  Arives  at  Designated  Coordinates 

00:20:00.00 

00:18:16.84 

00:19:34.86 

00:18:58.48 

00:00:20.44 

100.00 

90.00 

Percent  Steps  Correct 

96.00 

0 

96.00 

This  DOES  meet  the  performance  criterion  of  90% 

Platoon  Initiates  Screen  Operation 

00:20:00.00 

00:17:56.41 

00:20:14.44 

00:19:04.15 

00:00:31.37 

92.00 

90.00 

Percent  Steps  Correct 

96.00 

0 

88.00 

This  does  NOT  meet  the  performance  criterion  of  90% 

Driver  Reacts  to  Ambushl 

00:10:00.00 

00:08:19.06 

00:10:09.37 

00:09:11.21 

00:00:30.59 

96.00 

90.00 

Percent  Steps  Correct 

92.00 

0 

88.00 

This  does  NOT  meet  the  performance  criterion  of  90% 

Vehicle  Commander  Reacts  to  Ambushl 

00:10:00.00 

00:07:56.72 

00:09:52.43 

00:09:00.04 

00:00:24.85 

100.00 

90.00 

Percent  Steps  Correct 

96.00 

0 

96.00 

This  DOES  meet  the  performance  criterion  of  90% 

Infantry  Squad  Reacts  to  Ambushl 

00:10:00.00 

00:08:17.82 

00:09:47.78 

00:08:50.08 

00:00:25.59 

100.00 

90.00 

Percent  Steps  Correct 

92.00 

0 

92.00 

This  DOES  meet  the  performance  criterion  of  90% 

Platoon  Leader  Reacts  to  Ambushl 

00:10:00.00 

00:08:02.94 

00:09:32.83 

00:08:52.93 

00:00:24.06 

100.00 

90.00 

Percent  Steps  Correct 

100.00 

0 

100.00 

This  DOES  meet  the  performance  criterion  of  90% 

Evacuate  Injured  Personnel  from  BFV1 

00:20:00.00 

00:17:34.35 

00:19:54.06 

00:18:50.51 

00:00:31.65 

100.00 

90.00 

Percent  Steps  Correct 

92.00 

0 

92.00 

This  DOES  meet  the  performance  criterion  of  90% 

Disengage  from  an  Enemy  Forcel 

00:20:00.00 

00:17:57.22 

00:20:03.71 

00:19:00.60 

00:00:36.99 

92.00 

90.00 

Percent  Steps  Correct 

76.00 

0 

68.00 

This  does  NOT  meet  the  performance  criterion  of  90% 

Treat  and  Evacuate  Casultiesl 

00:20:00.00 

00:17:36.50 

00:19:50.97 

00:19:05.12 

00:00:33.61 

100.00 

90.00 

Percent  Steps  Correct 

92.00 

0 

92.00 

This  DOES  meet  the  performance  criterion  of  90% 

Conduct  Resupply  Operationsl 

00:20:00.00 

00:18:23.12 

00:19:57.86 

00:19:03.19 

00:00:28.29 

100.00 

90.00 

Percent  Steps  Correct 

92.00 

0 

92.00 

This  DOES  meet  the  performance  criterion  of  90% 

Conduct  Maintenance  Operationsl 

00:20:00.00 

00:17:37.41 

00:19:45.42 

00:18:54.63 

00:00:27.62 

100.00 

90.00 

Percent  Steps  Correct 

96.00 

0 

96.00 

This  DOES  meet  the  performance  criterion  of  90% 

Conduct  Consolidation  and  Reorganization! 

00:20:00.00 

00:17:47.45 

00:20:06.05 

00:19:01.20 

00:00:32.18 

92.59 

90.00 

Percent  Steps  Correct 

92.59 

0 

85.19 

This  does  NOT  meet  the  performance  criterion  of  90% 

Destroy  Unit  Vehicles  and  Equipment2 

00:20:00.00 

00:17:55.55 

00:19:38.60 

00:18:52.68 

00:00:24.30 

100.00 

90.00 

Percent  Steps  Correct 

96.00 

0 

96.00 

This  DOES  meet  the  performance  criterion  of  90% 

Resume  Original  Mission2 

00:20:00.00 

00:18:10.37 

00:20:05.01 

00:18:59.88 

00:00:30.03 

96.00 

90.00 

Percent  Steps  Correct 

100.00 

0 

96.00 

This  DOES  meet  the  performance  criterion  of  90% 

END 

00:07:00.00 

00:04:51.16 

00:06:36.74 

00:05:54.19 

00:00:26.88 

100.00 

90.00 

Percent  Steps  Correct 

96.00 

0 

96.00 

This  DOES  meet  the  performance  criterion  of  90% 

Mission  Success  Rates 


Run 

Mission  Performance  Time 

RNS 

Accuracy  Result 

1 

05:38:13.78 

II 

flPRTNT 

2 

05:38:11.31 

3 

05:36:48.93 

4 

No  failure 

4 

05:39:56.18 

5 

No  failure 

5 

05:40:36.60 

6 

No  failure 

6 

05:41:00.88 

7 

No  failure 

7 

05:39:12.05 

8 

No  failure 

8 

05:36:08.66 

9 

No  failure 

9 

05:40:42.82 

10 

No  failure 

10 

05:39:38.28 

11 

No  failure 

11 

05:39:10.11 

12 

No  failure 

12 

05:34:43.11 

13 

No  failure 

13 

05:38:48.06 

14 

No  failure 

14 

05:39:44.32 

15 

No  failure 

15 

05:35:47.95 

16 

No  failure 

16 

05:55:01.82 

17 

No  failure 

17 

05:37:18.57 

18 

No  failure 

18 

05:38:39.54 

19 

No  failure 

19 

05:40:10.08 

20 

No  failure 

20 

05:38:52.16 

21 

No  failure 

21 

05:35:53.98 

22 

No  failure 

22 

06:00:09.75 

23 

No  failure 

23 

05:36:05.52 

24 

No  failure 

24 

05:38:45.16 

25 

No  failure 

25 

05:37:21.77 

26 

No  failure 

HSJ 
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Simulation  Results 


srm/AR 

'im 


▼  Simulation  output  identified  tasks  that  did  not  meet  the 
Future  Combat  System  accuracy  standard. 

▼  IMPRINT  outputs  of  operator  workload  and  resource 
conflicts  were  further  investigated  to  determine  if  an 
overloaded  condition  or  a  resource  shortage  contributed 
to  the  accuracy  detriment  of  the  tasks. 

▼  Analysis  verified  that  the  Human  View  static  products  can 
be  used  to  structure  the  input  data  to  a  simulation  tool, 
such  as  IMPRINT,  to  provide  the  simulation  environment 
for  the  dynamic  Human  View. 

▼  The  dynamic  Human  View  is  critical  in  the  architecture 

framework  approach  because  it  captures  the  dynamic 
aspects  of  the  human  system  components  defined  in  other 
views.  H&l 
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SPAWAR 


Human  View  Products  Supported  by 
Modeling  aridSimulation  Example 


Human  Views 
supported  by 
IMPRINT 


performs 


Input 

Variable 


OUTPUT  -  Crew  Performance 
OUTPUT-  Crew  Workload 


Input 

Variable 


OUTPUT  -  Manpower 
Requirements 


T 


Task 

Task  -  Role 
Assignment 

r> 

responsible 

£> 


Manpower 

Output 

Establishment 

Variable 

i 

Role 

Depend 

L  encies  L 

Role 

u 

coordinate 

System 

Resource 


Task- 

System 

Interfaces 


KSA  to 
Training 
Mapping 


Input 

Variable 
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Comments  - 1 


T  Several  efforts  in  various  countries  are  underway  to 
define  and  structure  Human  View  as  viable 
methodology  for  HSI  practitioners  to  coordinate  and 
collaborate  with  the  system  engineers. 

[Example:  UK  MODAF  Human  View] 

T  While  the  ergonomists  always  had  a  set  of  tools  and 
processes  to  support  system  development  (e.g.,  task 
analysis,  function  allocation,  etc.),  the  Human  View 
products  facilitate  a  more  structured  language  for 
communicating  with  the  other  engineering  disciplines 
during  system  development. 

HSJ 
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Comments  -  2 


T  The  Human  View  products  are  derived  using  an 
ergonomic  approach,  namely,  a  top  down  method 
analyzing  human  gaps  in  existing  architecture 
frameworks,  or  based  on  specific  needs  that  evolved 
during  the  course  of  the  architecture  development  to 
capture  specific  human  view  data. 

T  HSI  practitioners  can  use  Human  View  methodology  to 
provide  a  fully  integrated  set  of  products  that  ensure 
an  effective  and  efficient  design,  development,  and 
production  process. 


HSJ 
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Conclusions 


▼  Human  View  products  facilitate  a  more  structured  language 
for  communicating  with  other  disciplines  during  system 
development. 

T  HSI  practitioners  can  use  Human  View  methodology  to 
provide  a  fully  integrated  set  of  products  that  ensure  an 
effective  and  efficient  design,  development,  and  production 
process. 

T  Analysis  results  demonstrated  that  Human  View  data  for  a 
complex  system,  such  as  the  Future  Combat  System,  can 
be  used  to  assess  design  impacts  when  combined  with  a 
simulation  tool,  such  as,  IMPRINT. 

▼  Dynamic  Human  View  is  critical  in  the  architecture 
framework  because  it  captures  the  dynamic  aspects  of  the 
human  system  components  defined  in  other  views.  KSJ 
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BAE  SYSTEMS 


50+  Years  Designing/Manufacturing  Over  6400  Recovery  Systems 


1951 :  Designed  the  M74,  First  Recovery  Vehicle  with 
Hydraulic  Powered  Recovery  Systems 

1953:  Contract  Award  to  convert  1,100  M4A3’s  to  M74  RV’s 
1954:  Designed  and  Prototyped  3  T-88  Recovery  vehicles 
1960:  Contract  Award  for  1 ,075  M88A0  Recovery  vehicles 
1963:  1,000  M88  Produced 

1965:  Production  of  1,844  M578  Light  Recovery  vehicles  begins 

1972:  875  Vehicle  M88  Conversion  to  Diesel  Contract 
Awarded 

1975:  M88A1 

1982:  M88AX  Automotive  Demonstrator  Design 

1987:  M88A1E1  Prototype  Contract 

1 993  5  M88A1  El ’s  Complete  Testing 

1994:  LRIP  Award  for  M88IRV  (M88A2  production  begins) 

1997:  M88A2 

2009:  491  M88A2  vehicles  delivered  to  customers  globally 

Today:  Over  2000  M88  vehicles  are  in  service 
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M88  Family  of  Vehicles  History 


Recovery  of  M48-M60  Tanks 

•  M88  Steel  Fabricated  Hull 

•  AVIS-1 790-6A,  12  Cylinder 
Petrol  Engine 

•  It  was  the  First  Armored 
Recovery  Vehicle  to  be 
Designed,  Produced,  and 
Fielded  as  a  New  System 

•  Mechanical  Transmission 


Baseline  Configuration 
Enhancements 


Recovery  of  M48-M60  Tanks 

•  M88  Steel  Fabricated  Hull 

•  AVDS-1790-2DR,  12  Cylinder 
Diesel  Engine  (750  HP) 

•  Increased  Operating  Range  from 
360  to  450  km 

•  Modified  Transmission 

•  Diesel  Fired  Personnel  Heater 

•  Auxiliary  Power  Unit 


Recovery  of  M1A1/M1 A2  Tanks 

•  M88  Steel  Fabricated  Hull 

•  AVDS-1790-8CR  12  Cylinder 
Diesel  Engine  (1050  HP) 

•  Recovery  for  70  Ton  MBT 

-  20-25%  Improved  Towing 

-  55%  Improved  Main  Winch 

-  40%  Improved  Hoist  winch 

•  Added  Auxiliary  Winch 

•  Improved  Ballistic  Protection 
•Laser  Protected  Vision  Blocks/Scope 
•Enhanced  Hydraulic  Diagnostics 
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BAE  SYSTEMS 


Schedule  Development 


•  What  is  the  most  popular  way  to  plan  schedules? 

•  SOW  to  Solution  Confusion 

•  Reverse  thinking  and  define  measurement  needs  first 

•  Is  EVMS,  CPI  and  SPI  required  and  supported  by  funding  or  will  something 
simpler  work? 

•  Defining  these  measures  can  help  indicate  depth  of  WBS  detail 

•  Then  dissect  the  SOW  to  develop  a  WBS  based  on  deliverables 

•  Socialize  with  your  IPT-  your  life  depends  on  it  -  train  as  needed  -  gain 
agreement  &  approval 

•  WBS  sets  the  stage  for  responsibilities 

•  Define  time  phased  sequence  of  events  including  internal  &  external 
reviews,  decision  gates  and  milestones  that  end  in  deliverables 

(referred  to  as  a  Summary  Schedule  or  Horse  Blanket  Schedule) 
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Schedule  Development  -  continued 

•  $1 B  Question 

•  How  deep  do  we  dive  to  reach  deliverables? 

•  What  does  the  customer  require  and  is  there  funding  to  support  it? 

•  What  is  the  size  and  complexity  of  the  project? 

•  Is  there  any  new  technology  development? 

•  Who  and  what  resources  are  needed  for  the  project? 

•  How  accurate  are  the  material  estimates? 

•  Do  we  need  activities  narrowed  down  to  the  point  of  selecting  a  bolt? 

•  How  urgent  is  the  need?  Do  they  need  it  tomorrow? 
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Continue  Schedule  Development 


•  The  answers  are  based  on  legal  requirements,  your  organizational 
requirements,  customer  requirements  and  the  risk  against  deliverables 

•  Create  a  draft  detailed  schedule 

•  Socialize  with  the  Project  Team,  Management  and  Customer  for 
agreement 

•  Measure,  monitor  and  continuously  update  the  schedule  methodology 
from  lessons  learned  throughout  execution 


Remember  -  all  the  answers  depend  on  the  requirements  and  risks. 
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Agenda 


•  Introduction 

•  Schedule  Development 


•  Requirements  Management 


•  Risk  Management 

•  Summary 
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Requirements  Management 


•  Requirements  for  legacy  systems  may  not  be  documented  in  typical 
Systems  Engineering  documentation  and  tools 

•  What  should  be  the  approach  to  defining/refining  requirements  for 
technical  support  projects  on  legacy  systems? 

•  Requirements  efforts  must  be  tailored  to  fit  the  size  and  cost  of  the 
project 

•  Establishing  a  requirements  baseline 

•  Clarifying  ambiguously  derived  requirements 


Successfully  tailored  requirements  process  helps  to  maintain 

budget,  scope,  and  performance 
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Requirements  Management  -  continued 


•  Example:  Developing  an  automated  hydraulic  diagnostics  on  the  M88A1 

•  Research  documentation  for  existing  requirements 

•  Statement  of  Work  (SOW) 

•  Purchase  Description 

•  Tech  Manuals 

•  Previous  milestone  review  materials 

•  Subject  Matter  Experts 

•  Any  potential  source  of  data 

•  Identify  legacy  system  (vehicle)  requirements  that  are  relevant  to  hydraulic 
diagnostics 

•  Operational  requirements 

•  Interface  requirements 

•  Baseline  requirements  determined  at  System  Requirements  Review  (SRR) 

•  Opportunity  for  customer  to  review  requirements  and  traceability 

•  Review  updates  to  requirements  at  following  milestone  reviews 
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Requirements  Management  -  continued 


•  Tailoring  Challenges 

•  Appropriate  level  of  requirements  for  the  scope  of  the  project 

•  Over-committing  can  lead  to  cost/schedule  overruns 

•  Useful  to  consider  verification/validation  matrix,  especially  when  addressing  system-level  requirements 

•  Modeling  and  simulation  may  be  limited  in  determining  capability  to  meet  requirements 

•  System-level  models  may  not  exist,  and  may  be  too  costly  to  develop  within  scope  of  the  project 

•  Cost-benefit  and  trade  studies  are  beneficial  in  resolving  conflicting  requirements,  especially  when 
legacy  technology  is  involved 

•  Robust  Risk  and  Opportunity  Management  is  critical  in  maintaining  cost  and  schedule 

•  Provides  a  structured  method  to  identify,  assess,  and  communicate  potential  issues  due  to  limited  scope 

•  Provides  a  structured  method  to  identify  performance  and  cost  benefits  which  may  be  realized  on  current 
projects  or  follow-on  projects 

•  Overall,  requirements  process  is  similar  to  “standard”  requirements  development 

•  Process  must  be  appropriately  tailored  to  fit  the  project 
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Agenda 


•  Introduction 

•  Schedule  Development 

•  Requirements  Management 

•  Risk  Management 

•  Summary 
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Risk  Management 
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We  all  do  risk  management  and 
mitigation,  even  if  not  formally 
documented. 

•  What  are  some  examples? 


How  to  manage  risks  at  a  level 
appropriate  to  the  project? 

Tailoring 


Risk  management  is  applicable  on  EVERY  project. 
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Risk  Management  -  continued 

Project  (Work  Directives)  Examples 


BAE  SYSTEMS 


•  Find  a  replacement  for  a  major  sub-assembly  in  the  vehicle. 

•  Customer  active  member  of  weekly  team  meetings 

•  Approx  $25M  in  funding 

•  Risk  approach  ->  Formal  risk  register  at  project  level,  team  involved  customer, 
costing  of  risks  tracked  separately,  met  at  a  minimum  monthly  to  update  status  of 
actions,  prioritized  based  on  probability  and  impact. 

•  Work  on  improvements  needed  to  bring  older  model  up  to  date. 

•  13  tasks  (develop,  integrate,  drawings,  maintenance  manuals,  etc)  under  main 
title  assigned  to  different  project  leads. 

•  Approx  $5M  in  funding 

•  Risk  approach  ->  Informal  risk  management  at  project/task  level,  risks  captured 
formally  in  Engineering  Program  risk  register,  customer  not  involved  in  risk  review 
board,  monthly  program  risk  meetings  to  update  status  of  actions. 
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Risk  Management  -  continued 


•  Establish  basic  requirements  for  all  projects: 

•  Statement  (If.. .due  to...,  then...) 

•  Handling  Approach  -  If  mitigation,  include  due  dates  and  assignee. 

•  Prioritization  method 

•  Periodic  reviews 

•  Contingency  plans 

•  Tailor  other  areas  of  risk  management: 

•  Reporting  frequency  and  structure 

•  Level  of  detail 

•  Costing  approach 

•  Project  risks  versus  Program  risks 
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Risk  Management  -  continued 


•  Best  approach  when  dealing  with  multiple  levels  of  tailoring 

•  Multiple  levels  of  management 
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A  -  Risk  registers 
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Agenda 


•  Introduction 

•  Schedule  Development 

•  Requirements  Management 

•  Risk  Management 

•  Summary 
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Summary 


Of  the  three  areas  covered,  you  can  see 
importance  of  understanding  the  customer  needs, 
cost,  and  type  of  each  project. 

•  Where  else  could  we  apply  this  approach? 

•  Verification  and  Validation 

•  Quantitative  Project  Management 


Just  as  you  need  a  good  CAST  when  you  go 
fishing  in  order  to  place  your  bait  in  the  stream  to 
catch  the  elusive  trout.... 

You  need  good  tailoring  of  the  Systems 
Engineering  processes  to  effectively  apply  the 
techniques  to  result  in  a  successful  project! 


So  when  you  start  a  project,  remember  it’s  all  about  the  CAST. 
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Summary  -  continued 


C  -  Identify  what  the  customer  wants! 

(timeframe  of  need,  level  of  participation,  etc) 

A  -  Assess  each  project  individually 

(same  approach  does  not  apply  to  all) 

S  -  Consider  the  size  or  funding  level 

(dollar  amount,  number  of  people,  number  of  tasks,  etc) 

T  -  Consider  the  type  of  project 

(development,  integration,  field  support,  reset,  etc) 


A  good  CAST  produces  accurate  tailoring  for  project  success. 
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Questions? 
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NATIONAL  DEFENSE  INDUSTRIAL  ASSOCIATION 


STRENGTH  THROL  (.11 INIH  STR^  &  TECHNOLOGY 


Producibility  M&S  Needs  for  Early 
Systems  Engineering  Evaluations  of 
Alternative  Design  Concepts 


Dr.  Al  Sanders 
October  29,  2009 


JCSEM  M&S  Sub-Committee 
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•  Joint  Committee  for  Systems  Engineering  &  Manufacturing 

-  Sponsored  by  NDIA  Systems  Engineering  &  Manufacturing  Divisions 

-  Chaired  by  Dr.  Tom  Christian  (SE)  and  Mike  Packer  (Manufacturing) 


•  JCSEM  M&S  Sub-Committee  Chartered  in  November  2008 

-  Dr.  Al  Sanders  -  Chairman  (Honeywell) 

-  John  Allen  (Honeywell) 

-  Kevin  Fischer  (Rockwell  Collins) 

-  Greg  Pollari  (Rockwell  Collins) 

-  Charlie  Stirk  (Cost  Vision) 

-  Dr.  Gary  Belie  (LMCO) 

-  Simon  Frechette  (NIST) 

-  Tim  Comerford  (Missouri  University) 

-  Scott  Frost  (Anser) 

-  Brench  Boden  (AFRL) 
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Early  Producibility  Focus  Motivation 


•  Early  decisions  responsible  for 
many  production  ramp  issues 

-  Actual  costs  exceed  estimates 

-  Quality  levels  below  expectations 

-  Low  yield  and  delivery  problems 

-  Service  and  sustainability  issues 

-  Integration  &  assembly  problems 

-  Overall  supply  chain  inefficiencies 

•  DoDI  5000.02  implemented  to  drive 
earlier  knowledge-based  decisions 

-  Increased  focus  on  SE  discipline 

-  Increased  focus  on  manufacturing 

-  Analysis-based  approaches  needed 

-  Producibility  most  neglected  “ility” 

-  Producibility  drives  hidden  costs 


New  5000.02  Approach 
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New  Approaches  Required  to  Address  Producibility  Risks 
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Current  State  of  Producibility  M&S 


•  Many  producibility  issues  driven 
by  early  SE  &  design  decisions 

-  Producibility  forgotten  requirement 

-  Producibility  hard  to  quantify  early 

-  Producibility  M&S  tools  immature 

•  Most  producibility  analyses  are 
CAD-based  rule  checkers 


Systems  Engineering  Process 


-  Require  nearly  final  design  layout 

-  Occur  too  late  to  influence  design 

-  Only  as  good  as  rules  loaded  in 

•  Need  quantitative  low  &  high- 
fidelity  tools  for  trade  studies 

-  Balance  performance/producibility 

-  Guide  analysis-based  decisions 

-  Shape  design  vs.  verify  problems 


Void  Exists  in  Current  Producibility  M&S  Capabilities 


JCSEM  Committee  Objective 
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•  Overall  JCSEM  Mission 

-  Integrate  manufacturing  and 
producibilty  considerations  into  early 
systems  engineering  activities 

•  Policy  sub-committee  charter 

-  Identify  and  update  key  SE  policy 
documents  to  drive  early  focus  on 
manufacturing  &  producibility 

•  People  sub-committee  charter 

-  Identify  critical  producibilty 
engineering  skills  required  for  early 
manufacturing  engagement  in  SE 

•  M&S  sub-committee  charter 

-  Identify  industry  M&S  analysis  needs 
required  to  address  producibility 
concerns  in  early  design  activities 
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Goal  is  to  Move  Manufacturing  to  the  Left  in  Acquisition 


JCSEM  M&S  Sub-Committee  Scope 
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“Identify  industry  M&S  analysis  needs  to  facilitate  the 
integration  of  producibility  concerns  into  the  earliest 
phases  of  the  system  engineering  process” 

In-Scope: 

•  Product  &  process  centric  analyses  to  guide  design  decisions 

•  Factory  &  supply  chain  analyses  to  guide  industrial  base  design 

•  Methodologies  to  integrate  producibility  into  SE  trade  studies 


Out-of-Scope: 

•  Virtual  collaboration  tools  and  enhancements  to  existing  software 

•  Data  standards,  protocols,  and  interoperability  requirements 

•  Digital/IT  type  solutions  to  facilitate  information  sharing 


Focus  is  Identifying  M&S  Needs  that  do  not  Exist  Today 


Sub-Committee  Technical  Approach 
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•  Objectives  and  Focus  Areas 

-  Identification  of  product,  process,  and  supply  chain  analysis  needs 

-  Identification  of  a  producibility  figure  of  merit  “goodness”  measure 

-  Identification  of  viable  approaches  for  SE  trade  study  integration 


•  Technical  Approach 

-  Identify  the  key  inputs  that  would  go  into  a  producibility  figure  of  merit 
calculation  to  capture  and  quantify  producibility  concerns 

-  Identify  specific  M&S  focus  areas  where  producibility  analysis 
capabilities  are  needed  to  support  system  design  activities 

-  Define  what  type  of  information  the  analyses  should  provide  at  each 
step  in  the  system  design  and  development  process 

-  Identify  potential  system  trade  study  approaches  that  enable 
producibility  evaluations  to  be  integrated  into  design  activities 


Goal  is  to  Provide  Investment  &  Implementation  Guidance 


Producibility  Figure  of  Merit  Elements 
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•  Producibility  definition  used  by  sub-committee: 

-  Producibility  defined  as  ease  and  economy  of  manufacturing  an  item,  or 
group  of  items,  in  large  quantities  in  a  production  environment 

-  Most  producibility  costs  “hidden”  in  nature  such  as  scrap,  rework, 
missed  deliveries,  safety  stock,  and  lead  time  buffers  due  to  low  yield 


Key  Factory  Metrics 

Producibility  Life  Cycle  Cost  Drivers 
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Weight  factors  would  be  assigned  to  each 
element  of  the  figure  of  merit  based  on 
relative  cost  impact  and  risk  for  critical 
systems,  sub-systems,  &  components 
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Manufacturing  Cost  Currently  Considered 
Manufacturing  Cost  not  Currently  Considered 
Hidden  Factory  Cost  not  Currently  Considered 


Figure  of  Merit  Links  Producibilty  to  Key  Factory  Metrics 
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Product  &  Process  Centric  Analyses 


Matrix  focus  areas: 

•  Should  cost  analyses 

•  Yield  prediction  models 

•  DFX  analyses 

•  Manuf  process  modeling 

•  Production  line  modeling 

•  Physics  based  analyses 
(casting,  solder  flow,  etc.) 

•  System  integration,  assembly, 
&  test  modeling 

•  Operator  assembly  &  test 
modeling,  e.g.,  ergonomics 

•  Obsolescence  modeling 


*  sensitivity  analysis  of  component  key 
characteristics  limits 

identification  of  critical  components  driving 
yield  fallout 

identification  of  design  simplification  and  yield 
improvement  opportunities 
prediction  of  component  variation  impact  on 
yield 

prediction  of  test  coverage  overlap  impact  due 
to  multiple  assembly  processes 
prediction  of  DFM  violation  impact  on  yield 
fallout 

prediction  of  yield  fallout  due  to  supplier 
process  capability  variability 
analysis  of  component  type  and  processing 
alternatives  on  yield 

analysis  of  component  yield  classification 
^reworkable^v^scra^ 


M&S  Analysis  Output  for  each  Design  Phase  Identified 


Supply  Chain  Design  &  Analyses 
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Analysis  focus  areas: 

•  Distribution  aspects 

-  Infrastructure  complexity 

-  Business  strategy  alignment 

-  Logistics/queuing  delays 

-  Environmental  events 

•  Technical  aspects 

-  Product  complexity 

-  Material  availability/maturity 

-  Process  learning  curves 

-  Technology  maturity 

-  Work  force  maturity 

-  Sustainability  impact 

-  Contract/policy  constraints 

-  Trend  analysis  &  diagnostics 


“System  Performance” 


System  Modeling  Approach  for  Industrial  Base  Design 
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Producibility  M&S  Linkage 


Product,  Process,  &  Supply  Chain  Producibility  Analysis  Tools 

Producibility  Life  Cycle  Cost  Drivers 

Should  Cost  Modeling 

Yield  Modeling 

DFX  Analyses 

Process  Modeling 

Production  Line  Modeling 
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Simulations 
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Weight  factors  would  be  assigned  to  each  element  of 
the  figure  of  merit  based  on  relative  cost  impact  and 
risk  for  critical  systems,  sub-systems,  &  components 


Producibility  Figure  of  Merit  Integrates  M&S  Tool  Output 
into  a  Single  “Goodness”  Measure  for  Trade  Evaluations 


SE  Trade  Study  Integration 
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•  Manufacturing  VOC  to  be  included  in  trade  study  process 

-  Responsible  for  long-term  production  of  the  proposed  system 

-  Provides  input  on  production  cost,  quality,  delivery,  &  inventory  goals 

-  Establishes  process  capability,  cycle  time,  and  yield  flow  down  targets 


•  Quality  Function  Deployment  (QFD)  based  methods 

-  Most  common  trade  study  tool  to  down  select  alternative  concepts 

-  Help  translate  customer  needs  into  system  specs  and  design  criteria 

-  Correlate  key  technical  performance  measures  to  acquisition  cost 

-  Mature  approach  that  can  be  easily  adapted  to  include  producibility 

•  Value  Driven  Design  (VDD)  based  methods 

-  Integrates  systems  engineering,  optimization,  and  economic  principles 

-  Leverages  requirements  flexibility,  optimization,  and  value  models 

-  Helps  balance  among  competing  TPM’s  to  produce  best  system  offering 

-  Emerging  research  area  that  addresses  limitations  of  QFD  approaches 


Producibility  M&S  Capability  Enables  Trade  Integration 


DoD  and  Industry  Benefits 


STRENGTH  THROt  (.11  INDl STR\  &  TECHNOLOGY 


•  Several  GAO  studies  conducted  around  acquisition  cost  overruns 

-  Systemic  issue  was  excessive  design,  technology,  &  manufacturing  risk 

-  Successful  programs  exhibited  earlier  design  &  producibility  knowledge 

-  Recommendation  is  adoption  of  knowledge-based  decision  processes 


•  Producibility  analysis  capability  generates  critical  knowledge  early 

-  Provides  means  to  influence  and  validate  requirements  feasibility 

-  Provides  means  to  identify,  quantify,  and  proactively  plan  for  risk 

-  Provides  manufacturing  analysis  capability  comparable  to  engineering 

•  Producibility  figure  of  merit  provides  means  to  quantify  concerns 

-  Provides  means  to  quantify  “hidden  costs”  during  early  design  studies 

-  Provides  means  to  guide  industrial  base  solutions  and  minimize  risk 

-  Provides  means  to  down  select  most  producible  design  alternatives 


Producibility  M&S  Enabler  for  Early  Knowledge 

Integration 


Summary  &  Recommendations 


STRENGTH  THROt  (.11  INDl STR\  &  TECHNOLOGY 


•  Producibility  is  neglected  “ility”  due  to  lack  of  analysis  capability 

-  Producibility  issues  are  difficult  to  predict  and  drive  “hidden”  costs 

-  Manufacturing  VOC  needs  to  be  included  in  requirements  definition 

-  SE  trade  studies  need  to  incorporate  producibility  considerations 


•  Producibility  M&S  is  a  critical  research  area  that  has  been  missing 

-  M&S  tools  required  to  drive  manufacturing  to  left  in  acquisition 

-  Product,  process,  &  supply  chain  centric  analyses  are  needed 

-  Requires  focused  research  attention  and  investments  to  mature 


•  Top  level  framework  established  for  SE  trade  study  integration 

-  Producibility  figure  of  merit  developed  as  “goodness”  measure 

-  Current  QFD-based  methods  can  be  extended  to  address  producibility 

-  More  research  is  needed  to  develop  and  mature  VDD-based  approaches 
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1 .  System  Overview 

Project  Overview 


Legacy  Problems 

•  Limited  Memory 

•  Difficult  to  add  new  features 

•  Maintained  multiple  code  baselines  to  support  different 
platforms 

•  Adds  more  testing  and  development 

•  Slow  Processors 

•  System  could  not  process  more  data 
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1 .  System  Overview 

Project  Overview  (Cont.) 


•  Project  Description 

•  New  Hardware 

•  Faster  processor 

•  More  memory 

•  Added  Ethernet 


•  Port  legacy  software 

•  Interrupt  system  to  polling  system 

•  Addition  of  a  partitioned  RTOS 


Motherboard 
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NEW 

PROCESSOR 


1 .  System  Overview 

Partitioned  Operating  System  (OS) 


Data 


£ 


Partition  1 


Partition  2 


1 


Partition  3 


POS 


POS 


POS 


Module  Operating  System 


•Only  one  partition  may  run  at  a  time 

•Data  can  move  between  partitions  when  defined 

by  OS 
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1 .  System  Overview 

Partitioned  OS  (Cont.) 


■  Partition  1 
Partition  2 

■  Partition  3 

■  Spare 

01  23456789  10 

Time  (ms) 


■  Partition  1 

■  Partition  3 

■  Spare 

01  23456789  10 

Time  (ms) 

•Different  schedules  allow  different  Partitions  to 
run  when  needed 
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1 .  System  Overview 

Internal  Interfaces 

*  Ports  -  Calls  through  the  OS 

*  Queuing  »  80  |js  for  read/write  access 

*  Sampling  «  50  ps  for  read/write  access 

•  Shared  Memory  -  Directly  accessible  by  the  partition 

*  »  1 0  ps  for  read/write  access 

*  Numbers  vary  based  on  hardware  or  the  RTOS 


Georgia  1 

Tech  M  OirasiMinSs 


Applications  in  Integrated  Diagnostics-  8 


1 .  System  Overview 

External  Interfaces 


Interface 

Speed 

Usage 

Ethernet 

10/100  Mbps 

Net  loading  code  into  RAM 

1553 

1  Mbps 

Communication  with  other  systems  and 
Instrumentation  Data 

RS-232 

115.2  Kbps 

Starting  a  net  load  and 

Default  printf 

RS-422 

9.6  Kbps 

Legacy  debug 
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2.  Motivation 

General  Debugging:  Single  Partition 


Partition  time  allotted:  1  ms 
Partition  time  used:  800us 

Partition  time  unused:  200|is 


Normal 

Run 


Partition  Program 


Debug  statements:  »  300  [is 


Debug 

Run 


Partition 

Program 

Debug  Statement 

Partition  Program 
(Cont.) 

K  1 

i 

Time  Allotted  for  Partition  to  Run  y 
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2.  Motivation 

General  Debugging:  Multiple  Partitions 


Debua  Data 


i 


Partition  1  ■  Partition  2  Partition  3 


Partitioned  Operating  System 


•Multiple  partitions  used  the  same  debug  media 
•Data  overwrites 
•Debug  stream  contention 
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2.  Motivation 

System  Performance 


•  System  limitations 

•  Processor/memory  utilization  during  normal  operation 

•  System  throughput 

•  Amount  of  system  inputs 

•  System  latency 

•  Response  time  to  system  inputs 

•  System  data  flow 

•  Understanding  how  information  gets  from  point  A  to 
point  B  within  system 
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3.  Problem  Statement 


•  Avoid  Uncertainty  Principle 

•  Latency  introduced  by  diagnostics  drastically 
affecting  system 

•  Provide  as  much  information  as  possible 

•  Introduce  as  little  system  interference  as  possible 

•  Provide  information  that  is  easy  for  user  to 
understand  and  analyze 

•  Scalable  for  future  use 
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4.  Approach 

Interface 

•Ethernet 

•High  bandwidth 
•PC  Graphical  User  Interface 
•Real  time  display 
•Bit/Byte  analyzer 
•Raw/Parsed/Filtered  Data 
•Storage  for  post-analysis 
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4.  Approach 

Rate 

•Internal  and  External  considerations 


•External  Design  Considerations 

•How  often  is  data  required  to  debug? 
•How  much  data  is  required  to  debug? 


Data  Sent 


t 

Data  Sent 


t 

Data  Sent 


Data  Sent 


t 

Data  Sent 
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4.  Approach 

Rate  (Cont.) 


Data  Sent  Data  Sent  Data  Sent  Data  Sent  Data  Sent 


Data  Created  Data  Created  Data  Created 


Less  debug  data  created  than  possible 
(Is  there  enough  data  to  debug?) 


Data  Sent  Data  Sent  Data  Sent  Data  Sent  Data  Sent 


Data  Created  Data  Created  Data  Created  Data  Created  Data  Created 


Too  much  debug  data  created 
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4.  Approach 

Diagnostics  Partition 


Data 


£ 


£ 


-Debug  Data- 


Partition  1  Partition  2  Partition  3  Diagnostics 


Partitioned  Operating  System 


Diagnostic  Schedule  for  Schedule  B 


Schedule  C 


0123456789  10 


■  Partition  1 

■  Partition  3 

■  Diagnostics 

■  Spare 


Time  (ms) 


Georgia  1 

Tech  n  OmsM&nSs 


Applications  in  Integrated  Diagnostics-  20 


4.  Approach 

Diagnostics  Application 


A 


PC  Application 


Partition  1 
Window 

Partition  2 
Window 

Partition  3 
Window 

Ethernet 


Data 


± 


-Debug  Data- 


Partition  1  H  Partition  2  ml  Partition  3  I  Diagnostics 


Partitioned  Operating  System 
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5.  Results 

Debugging  with  Diagnostics 


•  Diagnostics  Debug 
statements  found  to  Nr™ 
take  =  10  ps 


Partition  Program 


•  Partition  allotted  1 
ms  to  run 


Normal 

Debug 

Run 


Partition 

Program 


Debug  Statement 


Partition  Program 
(Cont.) 


Diagnostics 

Debug 

Run 


Partition 

C 

CD  CD 

E  g 

Partition  Program 

Program 

CD  £ 

Q  eg 

(Cont.) 

CO 

Time  Allotted  for  Partition  to  Run 
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5.  Results 


•  Found  bugs  during  overnight  test  cases 

•  Processor  utilization  spikes  in  overnight  test  cases 

•  Queue  trickling  and  data  buffer  overflows 

•  Other  general  diagnostic  data  during  normal 
operation 

•  Possibilities  for  optimization 

•  Requirements  verification 

•  Seeing  the  inner  workings  of  the  system  with  limited 
system  interference 
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System  Interference  (%) 


5.  Results 


System  Interference  vs  Diagnostic  Throughput 


-•-1553 

^^Ethernet 

RS232 


100 

80 

60 


Diagnostic  Throughput 
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6.  Conclusions 


•  Lessons  learned 

•  Keep  interface  simple  for  ease  of  use 

•  Make  Ethernet  output  multicast  or  UDP 

•  Future  ideas 

•  Move  Diagnostics  partition  to  an  RTOS  Task  to  reduce 
latency  and  increase  throughput 

•  Make  the  interface  for  partitions  more  abstract  for 
scalability 

•  Work  with  developers  and  testers  for  more  synergy  in 
using  tool 
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6.  Future  Approach 


Disclaimer 


The  Opinions  Expressed  by  the  Speaker 

Are  His  Own 
and 

Do  Not  Necessarily  Reflect  Anyone  Else’s 

..although 
They  Might! 


The  Power  of  the  Spec: 

The  Many  and  Diverse  Roles  of 
Specifications  &  Standards  in 
Systems  Engineering 


Robert  B.  “Scott”  Kuhnen 
HQ  AFMC  Command  Stdzn  Office 
Wright-Patterson  AFB  OH 
28  October  2009 


First... some  table  setting. 

deja  vu  for  any  of  you? 

Any  of  you  here  in  2007? 
Any  of  you  remember. . . 


..this  outfit? 


How  to  Paint  a  Room 


The  Role  of  Specs  &  Standards  in 
the  Systems  Engineering... 

..Business! 


Robert  B.  “Scott”  Kuhnen 
ASC/AFRL  Eng  Stds  Office 
Wright-Patterson  AFB  OH 
24  October  2007 


Shall  We  Get  Started? 


Not  so  fast!!! 


“Proper  interior  paint  preparation  of  your 
walls  and  ceilings  before  painting  will  often 
encompass  more  work  than  the  actual 
painting.  Up  to  75%  of  the  work  can  be 
getting  a  surface  ready  for  painting.” 

Karl  Crowder 

http://www.house-painting-info.com/index.htnnl 


Tools  for  Prepping  Walls 


•  Safety  glasses  or  goggles 

•  Respirator  or  face  mask 

•  Ear  protectors 

•  Rubber  gloves 

•  Pry  bar 

•  Paint  scraper 

•  Wallpaper  steamer  (rent  if  needed) 

•  Can  opener  or  widening  tool 

•  Fan 

•  Hand  sanding  block 

•  Orbital  sander 

•  Screwdriver 

•  Putty  knife 

•  Sponge 

•  Cap  or  scarf 

•  Old  clothes 


Materials  for  Prepping  Walls 


Spackle  (compound) 
Fine-grit  sandpaper 

-  (100  -  120-grit  silicon  carbide) 


Detergent  and  ammonia  or  tri-sodium 
phosphate  (TSP) 

Self-adhesive  drywall  tape 

Primer  or  adhesive  pad 

Sizing  (for  wallpapering) 


Tools  for  Painting 


Drop  cloths 
Ladders 
Buckets 
Paint  edger 
Brushes,  4",  3",  and  11/2" 

Angled  sash  brushes,  1  1/2"  and  2" 
Roller  pan  with  screen 
Roller  covers  with  appropriate  naps 
Roller  handle 
Roller  extender 
Paint  guide 


Materials  for  Painting 


Masking  tape,  2"  wide 
Newspaper 

Adhesive  pad  or  primer 
Paint  thinner  (with  oil-based  paints) 
Aluminum  foil 
Rags 


Home 

DUMHIe3 


6 
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What  the  experts  say... 


•  Most  people  think  they  know  how  to  paint,  and  usually  the 
results  are  pretty  good.  But  for  painting  contractor  John 

Dee,  "pretty  good"  isn't  good  enough.  After  nearly  three  'N 

decades  of  rolling,  brushing,  and  spraying  paint  he  knows 
the  subtle  tricks  for  applying  smooth,  even  coats  to  walls, 
ceilings,  and  woodwork,  and  for  creating  crisp  boundaries 
between  colors. 


According  to  Dee,  there's  no  magic  to  getting  professional¬ 
looking  results.  Prar.tir.fi  hnlps,  and  thorough  siirfar.F; _ _ 

preparation  is  essential.  But  the  key,  he  says,  is  to  paint  in 
an  orderly,  systematic  way.  So  whether  he's  painting  a 
multi-paneled  door  or  a  flat  expanse  of  wall,  he  proceeds 
almost  scientifically  from  one  step  to  the  next,  with  no 
shortcuts.  "Your  approach  to  the  task,  the  order  in  which  yoi 1/ 

do  things,  can  speed  the  work  or  slow  you  down,"  Dee  says. 


ENGINEERING  DESIGN  PROCESS 


PERFORMANCE 

BASED 

REQUIREMENTS 


PRODUCT 
VERIFICATION  & 
ACCEPTANCE 


System 

Rqm’ts 

System  Qual  &  Prod 
Accpt  Criteria 


A  Theory  to  Live  By... 


Preparing  the  surface  is  the  most 
important  part  of  any  painting  project.  If 
the  paint  doesn’t  have  a  smooth,  clean 
surface  to  adhere  to, _the  result  will  be  a 
poor-quality  job  that  doesn’tTast  very  !on< 
“You  should  spend  at  least  as  much  time 
on  surface  prep  as  you  will  be  painting,” 
advises  Horst. 


If  it's  worth  doing,  it's  worth  doing  rjcjht  the  first 


time. 


Few  of 


us  really  realize  this,  or  even  like  to  admit  it, 
since  it  leads  to  more  work.  It  is  a  step  that  is  all 
too  often  left  out,  and  the  final  job  reflects  its _ 

^omission.  It  is  too  easy  just  to  start  painting  and  ^ 

not  go  through  the  necessary  prep  steps. 

Indeed,  for  a  while  the  paint  job  may  even  look 
pretty  good.  But  sooner  or  later  the  poor  quality 
will  show  up. 

_ 


Talking  about  painting  or...SE? 


HGUKL3  The  SV knits  Engineering  Process 


[nire  me  n  rs  A  n  alvsis  ^ 

1  Miltons  &  E el-1; : rci u di e el t '; 

IdeoriNFuac  no  ual  Requi reuientv.  - 
Define  ■RejWPerToriiL'iuce  &. 

De-nga  C  ota^E^HU  Rggiumnencs  j 

rfc  Loop 


Systems  Analysis  \ 

&  Control 
(Balance)  / 

+  Select  Fretenred 


/ 


F  n]ic  rional  Ana  lysis,*  Alio  cation 
+  Decompose  in  Lowtr-Ltvifl  Fuctioas 
+  Allocite  Performance  A  QtberLfiiuring 
Requirement  to  All  FunnaonaL  Leveh 
■  Define  Refitae  Functional  Interface;  (InjinmaUEiteraal) 
DefinB  REfioe  Inte^ratE  Functional  Architecture 


! 


Alternatives 
Trade -Off  Studies 

*  EffEclivene^  Analysts 

*  Rid;  Management 

C  oufigurafion  Minawmmt 

■  Interface  Management 

*  Data  Management 

■  PErTotmauce-Ba^ed 
PrOgreL':-  MEn.MJT¥QLeilt 
..  SEMS 

■■  TFM 

■■  Technical  Relies 


Design  Loop 


Synthesis 

Transform  Architecture*  (Functional  to  Fkyscal) 
D&fiiie  Alternative  System  ■C-uncepk, 

Configuration  Hem*  &  System  Element* 

Define  Refine  Physical  Interfaces  rlnternnlErfenial} 
Defiue  Alternative  Product  &  Proce*;  Solution; 


PROCESS  OUTPUT 


■  Decision  Data  Ba;e 

■■  Derifion  Support  Data 
■■  System  Func  tin  ual  &  Physical 
iIiiMiiIbii  i 

■■  Specification*  &  Eveline; 

■  Balanced  System  Solution* 


XlVMt  3661^1^11*1 


•  Defense  Specifications 

•  Defense  Standards 


•  Qualified  Products  Lists 

•  Non-Gov’t  Standards 

•  Int’l  Standards 


•  etc. 


- 
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F 


- 

= 

■: 
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PROCESS  INPUT 

1  CujtnuLer  Need^  OhjEctive-. 
REqELtemeuft 
**  Mi^s-loes. 

11  Meniure;  of  EffectiveEE^ 
■■  EnvLroninEttt; 

11  CoEitraiafi 

*  TEcliuoLo“y  Ea^e 

*  Prior  Output 
■  Program  DechLott 

REqEueineuti 

*  RequireiEent':.  From  / 
Tailored  Specification  / 

nud  Standards  ■' 


Require  me  d  f  s  A  n  alvsis  ^ 

Analyze  Mkaxtus  &  EnviroEuientv 
Ideudh'FuDcriimaL  Requirements  - 
Define  "RefinE  Performa  uce  A 
V  Deia  C  Msjnint  Requirements  J 


/  Systems  Analysis 
&  Control 
(Balance) 


I* 


Rf  (]J  i]  ELTQEE[^  Loop 


/ 


F  unc  rional  Analysis/ Aflo  cation 
Decompose  tn  Lomr-LETOl  Functions 
Allocate  Performance  &  Other  Lfinifin®: 

Requirements  tn  AH  Functional  Leveh 
Define  Refine  Functional  Interface-.  iluteinal  Ertenial} 
PefinB  Refine  Integrate  FnnctionaL  Architecture  > 

Design  Loop 


select  Preferred 
Alternatives 
Trade -Off  Studies 
Effectiveness  Aoaly^es 
Risk  Management 
C  nufiguration  Mann-getDEEt 
Interface  Management 
Data  Management 
PErTotrnauce-Based 
Progress  Measurement 
■■  SEMS 
■■  TPM 

■■  Technical  Reiiervs 


Verification 


Synthesis 

Transform  AnkihdErn  (Functional  tn  P  by  deal) 
Define  Alternative  System  Concepts, 

Configuration  Items  &  System  Elements 

Define  "Refine  Pliysical  Interfaces  (Internal'EitEnial) 

Define  Alternative  Product  &  Process  Solutions 


PROCESS  OUTPUT 


■  Decision  Data  Ba;e 
■■  Deridon  Support  Data 
■■  System  Ennc  tin  ual  &  Physical 
Architectures 

■■  Specifications  &  Baseline; 

1  Balanced  System  Solutions 
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DoD  Systems  Engineering  Shortfalls* 


•  Root  cause  of  failures  on  programs  include: 

-  Inadequate  understanding  of  requirements 

-  Lack  of  systems  engineering  discipline,  authority, 
and  resources 

t  > 

-  Lack  of  technical  planning  and  oversight 

-  Stovepipe  developments  with  late  integration 

-  Lack  of  subject  matter  expertise 

-  Availability  of  systems  integration  facilities 

-  Low  visibility  of  software  risk _ 

-  Technology  maturity  overestimated 


Major  contributors  to  poor  program  performance 


*  DoD-directed  Studies/Reviews 


Could  the  problem  be. 
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Perry  scraps  mil-specs 


have  not  been  clarified" 

The  directive,  which  will  he 
phased  in  over  the  next  six 
months,  mandates  that  ail  DoO 
procurement  con  | 
tracts  use  commatid 

and  industrial  specs 
and  standard*  where 
they  exist,  the  use  of 
mil -specs  will  mjuirv 
j  waiver  Radiation 

Secretary  of  Defense 
William  Perry  has  intredeettf 
far  reacfeiog  eftangas  to  the 
procurement  process 
including  mandating  the 
use  of  performance  specs. 

Iiardc-ncd  component* 
a tt  exempt  from  the 
directive. 

in  another  major 
change,  program  man 
awers  in*  now  required 
to  adopt  prrtnmiantr 
based  specific  at  ion* 
for  new  system*  and 


major  modifications  The  pertor 
manev  specs  describe  how  a  svx 
tem  »  to  function  bur  leave*  the 
iCofUmueJ  on  puna  32} 


Hv  Rntu;  Raynrr 


Washington,  DC —In  Lite  June, 
Defense  Secretary  William  rVriy 
ordered  a  dramatic  about  face  in 
the  Defame  Department's  use  nf 
military  specification*  and  stan 
dards  ordering  that  all  L>iD  pro 
gram*  rely  more  heavily  on  com¬ 
mercial  parts  and  adopt  a  perfor¬ 
mance  ha*cd  specification  process 
While  Perry  *  announcement 
was  widely  anticipated  and  pub¬ 
licity  applmded  by  the  defense  due 
tnwics  industry,  many  company 
officials  arc  concerned  that  tht 
changes  wifi  increase  uncertainty 
in  the  acquisition  prwcos  and 
threaten  some  existing  systems 
that  arc  operating  writ  such  as  tile 
Qualified  Manufacturing  Line 
;<)MI4,  a  DotVgptxlhq  system  fur 
certifying  a  manufacturing  process 
"Right  now  it  is  a  wait-and-see 
game."  says  Braid  Paulsen,  director 
Of  marketing  for  military  and 
aerospace  products  at  National 
Semiconductor  .Santa  Clara,  CAh 
'There  arc  a  Jot  of  issue*  that 


Summary  of  the  Weapon  Systems 
Acquisition  Reform  Act  of  2009: 


•  Section  101.  Systems  Engineering  Capabilities.  The  Defense  Science 
Board  Task  Force  on  Developmental  Test  and  Evaluation  reported  in 
r  May  2008  that  “the  single  most  important  step  necessary”  to  address  ^ 
high  rates  of  failure  on  defense  acquisition  programs  is  “a  viable 
^  systems  engineering  strategy  from  the  beginning.”  The  Government  J 


Accountability  Office  has  reached  similar  conclusions.  Unfortunately, 
the  Committee  on  Pre-Milestone  A  and  Early-Phase  Systems 
Engineering  of  Air  Force  Studies  Board  of  the  National  Research  Council 
reported  in  February  2008  that  the  Air  Force  has  systematically 
dismantled  its  systems  engineering  organizations  and  capabilities  over 
the  last  twenty  years.  The  other  services  have  done  the  same.  Section 

/^101  would  address  this  problem  by  requiring  DOD  to:  (1)  assess  the  ^ 

extent  to  which  the  Department  has  in  place  the  systems  engineering 
capabilities  needed  to  ensure  that  key  acquisition  decisions  are 
supported  by  a  rigorous  systems  analysis  and  systems  engineering 
process;  and  (2)  establish  organizations  and  develop  skilled  employees 
Reeded  to  fill  any  gaps  in  such  capabilities. _ 


Summary  of  the  Weapon  Systems 
Acquisition  Reform  Act  of  2009: 

•  Section  103.  Technological  Maturity  Assessments.  For  years  now,  the 
Government  Accountability  Office  (GAO)  has  reported  that  successful 
commercial  firms  use  a  “knowledge-based”  product  development  process  to 
introduce  new  products.  Although  DOD  acquisition  policy  embraces  this 
concept,  requiring  that  technologies  be  demonstrated  in  a  relevant 
environment  prior  to  program  initiation,  the  Department  continues  to  fall  short 

/of  this  goal.  Last  Spring,  GAO  reviewed  72  of  DOD’s  95  major  defense 
acquisition  programs  (MDAPs)  and  reported  that  64  of  the  72  fell  short  of  the 
required  level  of  product  knowledge.  According  to  GAO,  164  of  the  356  critical 
technologies  on  these  programs  failed  to  meet  even  the  minimum 
requirements  for  technological  maturity.  Section  103  would  address  this 
problem  by  making  it  the  responsibility  of  the  Director  of  Defense  Research 
and  Engineering  (DDR&E)  to  periodically  review  and  assess  the  technological 
\maturity  of  critical  technologies  used  in  MDAPs.  The  DDR&E’s  determinations^/ 

would  serve  as  a  basis  for  determining  whether  a  program  is  ready  to  enter 
the  acquisition  process. 


Summary  of  the  Weapon  Systems 
Acquisition  Reform  Act  of  2009: 

Section  206.  Acquisition  Excellence.  The  Department  o£^ 
Defense  will  need  an  infusion  of  highly  skiR^^|>cf|J0rer 
acquisition  specialists  to  ^rt\uajte^hg^^rrements  of  this  bill 
and  addressJhe^r^g©^)' we  defense  acquisition  system. 
3IhegC^\^tre'eirras  already  established  an  acquisition 
u  workforce  development  fund  to  provide  the  resources  needed  to 
hire  and  retain  new  workers. 

However,  positive  motivation  is  needed  as  much  as  money. 
Section  206  would  address  this  issue  by  establishing  an  annual 
awards  program  -  modeled  on  thaDemrtrlft^S accessful 
environmental  awardf^rWeft -Jck^IcOTpze  individuals  and 
teams  v\*io«  refclnr  contributions  to  the  improved  cost, 

schedul^^nS  performance  of  defense  acquisition  programs. 


Systems  Engineering 
Fundamentals  from  Past  Programs 


SE  was 
-  Syste 


s  conducted  by  the  des 

3IT^C  toft*  • 

ufjtwJilny  designs  tf1t?lg|(^HuliTimates 


marwpr(^^cjygriiBn\r  years 

CommoliMTOracteristics:  yesterday  and  today 

-  Small,  efficient  systems  engineering  staff 

•  Previous  design  engineers 

-  Knack  for  requirements 

^"Appreciated  the  larger  challerTye  at  the  system  level 

-  Not  always  collocated  and  not  always  the  same 
company 


Source:  Mr.  John  Griffin, 
former  ASC/EN  Director 


Technical  Wisdom  From  Our  Past . . . 


Joint  Service  Specification 

Guides 


JSSG-2000 


Air  System 


JSSG-2004 

Weapons 


JSSG-2005 

Avionics 


JSSG-2006 

Structures 


JSSG-2001 
Air  Vehicle 


JSSG-2007 

Engines 


JSSG-2002 

Training 


JSSG-2003 
Support  Sys 


JSSG-2008  JSSG-2009 

Vehicle  Control  Vehicle 
&  Mgmt  Subsystems 


JSSG-2010 

Crew 

Systems 


Unintended  consequences? 


A  PubHohof 


Alilitaiy&leiospace 

electronics 
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Perry  scraps  mil-specs 


WAMBNGTON,  DC~hi  Lite  lunc. 
Defense  Secretary  William  rVrty 
ordered  a  dramatic  about  face  in 
the  Defame  Department  *  ussc  ci 
military  ipecificatiom  ami  *tan 
dardk  ordering  that  all  L>iD  pn> 
gram*  rely  more  heavdy  on  com 
men  tal  pans  and  adopt  a  perior 
nunte  ha*cd  specification  process 
While  Perry's  announcement 
was  widely  anticipated  and  pub- 
lidy  applauded  by  the  dden.se  clue 
tropic*  industry,  many  company 
(Petals  are  concerned  that  the 
change  will  tncre.ise  unterimmy 
in  tbe  acquisition  process  and 
threaten  some  existing  systems 
that  arc  operating  well,  such  as  tile 
uallftcd  Manufacturing  Line 
MU,  a  L)uO-spixific  system  for 
ifymg  a  manufacturing  proem. 
Hight  now  it  \s  a  wait-and-see 
says  Brail  l»aulsen,  director 
ting  for  military  and 
auuj^vt  products  at  National 
Semiconductor  .Santa  C.Ura,  CM 
'There  arc  a  lot  of  issues  that 


have  not  been  clarified” 

The  directive,  which  will  he 
phased  in  over  the  next  six 
months,  mandate*  that  all  Dot) 


procurement  con 
tracts  use  commercial 
and  industrial  specs 
and  standard*  where 
they  exist,  the  use  of 
mil  aspect  will  mjiurr 
a  waiver  Radiation 


major  modjhcatmn*  The  pertor 
mauev  spec*  denenbe  how  a  «vs 
tern  »  to  function  hut  leaves  the 

(Contmued  an  /*jg*  32) 


Secretary  of  Defense 
WSNmi  Perry  has  initedeced 
far  reaching  changes  to  the 
procurement  process, 
including  mandating  the 
use  of  performance  specs. 


hardened  components 
ate  exempt  from  the 
directive. 

ip  another  major 
change,  program  man 
agETK  arc  now  required 
to  adopt  performance 
based  specification* 
for  new  system*  and 


Truth  is... we  noticed  issues  almost  immediately! 


The  Great  Challenge 


Humpty  Dumpty  sat  on  a  wall, 
Humpty  Dumpty  had  a  great  fall. 
All  the  king's  horses, 

And  all  the  king's  men, 
Couldn't  put  Humpty  together  again. 


EN  Technical  Processes 


Advanced  Tech  — 
Transition 

Requirements  Definition 


Integrated  Risk  Management 
Modeling  and  Simulation 


SOO/RFP 

O 


Allocation 

Contract 

o 


Configuration  Management 

Verification  “ 


EIRT 


Mod 

Concept 

Program  Definition  & 

Engineering  and  Manufacturing 

Production,  Fielding/Deployment 

Planning 

Exploration 

Risk  Reduction 

Development 

&  Operational  Support 

Disposal 

Integrity 

Programs 
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Operational  Safety,  Suitability  &  Effectiveness  Assurance 


♦ 


Critical  Safety  Item 

Joint  Instruction 


Exij&hcteci 

1 

f 

J 

~ - * 

AirwOfUlirieSS 

Mil-Hdbk-1798 

MH-Hdbk-1783: 

Mil-Hdbk-fl7244 

Catena 

MECSIP 

EN$IP 

AVIP 

_ A _ 3 

r 

- 1 

l - 

l 

r 

I 

Mii-StD  for 

A/W  Criteria 

Mil-$W-1530 

Mil-Sld  tor 
MECSIP 

Mii-Std  for 
ENStP 

Mii-Std  tor 
AVIP 

Existing 

f 

Pending 

Excerpts  from: 

Pre-Milestone  A  and  Early-Phase 
Systems  Engineering 

A  Retrospective  Review  and  Benefits  for 
Future  Air  Force  Systems  Acquisition 

“Two  critical  factors  in  the  success  or  failure  of  programs  that  fall  in  the 
latter  category  are  the  need  for  high-quality  systems  engineering  and  the 
related  issue  of  the  need  for  a  high-quality  systems  engineering  workforce.” 


“On  a  more  technical  level,  the  NRO,  in  cooperation  with  its  industry  team 
members,  has  reinstituted  a  minimum  essential  set  of  specifications  and 
standards  on  such  diverse  topics  as  systems  engineering  (SE)  and  the 
^  qualification  of  key  components.” 

“But  in  one  respect  the  complexity  of  most  large  systems  today  seems  to  be 
much  greater,  and  that  is  in  the  complexity  of  the  missions  that  the  systems 
are  asked  to  serve  and  in  the  number  and  diversity  of  users,  supporters,  and 
administrators  of  the  systems.  Indeed,  it  is  often  the  increased  complexity  of 
external  interfaces,  more  than  internal  system  design  complexity,  that  is  the 
cause  of  extended  development  times  and  costs.” 


Space  & 
Missile 
Center 
(SMC) 
..took  the 
first 

concrete 
steps... 
circa  2003! 


DEPARTMENT  OFTHE  AIR  FORCE 

H=.'nDOHyV=rCR5  ArtLl  M-SSiLh  SYSTEMS  CENTER  (/> FEFCj 

I  03  A-JflE-l  Alrt  FORCE  BASE  CALTOINIA 


MEMORANDUM  KO.R  SMC -ALL 
FROM:  SMC/CC 

SUBJECT:  Policy  L-trcr  on  Spccific^tEOri  and  Standard:?  TJ.^ge  d  SMC 

1 .  BHckgiDund:  A  key  element  ofiJie  Systems  Engineering  Reviulizatioa effort  is  the 
use  of  specifications  and  standards  as  pail  of  the  technical  baseline  of  the  SMC 
acquisition  process.  Prior  to  acquisition  reform,  usistff  military  specifi nations  and 
standards  in  Request  for  Proposals  (Rh'P],  contracts  and  program  min^cTntni  practices 
were  one  of  the  primary  mclhud&'ajqirDaEticK  used  Lo  define  technical  leqnireitieTiLsi. 

1 1 ■.image  contractor  performance,  mid  incorporate  signi  Hearn.  lessons  learned.  One  key 
element  of  acquisition  reform  was  to  el  Emirate  the  government  irom  contractually 
dictating  prescriptive  “how-to"  instructions  nr  processes  used  by  contractors.  For  a 
decade  we  have  limited  and  reduced  o  ar  use  ofspedflGations  and  standards  in  IthFs, 
proposal  evaluations,  contractor  performance  assessments,  and  on  contracts  aa 
compliance  doGumcnte.  The  unintet jtionsl  resu It  wji s  that  technic ul  baseline*  and 
pmutHHC*  were  compromised.  With  Lke  Lllfi lover.  Curtsulididioiis.  FJf.d  rctiretncnt  0  Litany 
industry  and  govommeni  personnel,  wc  have  hampered  our  ability  to  prtz  on  lessons 
learned  from  general  ion  In  generation. 

2.  This  directive  outlmes  the  framework  for  the  use  of  specifications  and  standards  as  an 
integral  element  of  SMC  acquisition,  contracting,  and  program  management.  There  is  no 
intent  to  return  to  the  pre  acquisiti  on  reform  approach  of  using  m  excessive  number  of 
qnei-.-ficatmns  and  standards;  and  prescribing  detailed  pnaM2iKe&.  A  list  uf  high-priori  tv 
criLiual  spyd  fi  caiitui  □  mid  n'.-u  islands  is  being  reviewed  and  established  feu;  appropriate  use 
ill  the  acqu i = it: oil  process .  TTii s  list  will  include  two  categories :  (1 )  those  that  contribute 
■O  mission  success  (areas  I  hat  caused  fail  i  ires,  ran  sod  ^ignifieanl  aunch  de  ays,  shortened 
.uissjon  life,  reduced  performance,  caused  excessive  rework;  or  generated  important 
lessons  learned)  ami  (2)  those  needed  tor  effective  program  :mplcrrcmUiLiim  (insight  in.tc- 
program  perfgmLUiiLe  or  status.  ri^k  reduction,  evaluations  and  analysis,  and  critical 
process  definitions j.  Hie  specifications  and  s'lcndands  selected  ior  the  technical  baseline 
will  hs  reviewed  in  Ugh:  of  currant  acquisition  practices.  Operational  Safety  Suitability 
md  Eflfcctivnwaa  policies,  and  new  tcehoicfll  knowledge.  they  will  bo  updated,  revised, 
and  tailored  as  anprooriate  .'or  use  at  SMC.  Sources  of  specifications,  and  standards  may 
Include  goveriunciLi,  industry,  previous  SMC  Conunandefs  Policies.  and  specifications 
and  standards  from  AIAAh  ISO.  or  other  professional  societies. 


C.UARDiAhL  UP  r--:  HlisH  FRCNIILI’. 


Specs  &  Standards 
Initiative 


I  THE  AEROSPACE 


CORPORATION 


Specs  &  Standards  Initiative 

*  Apply  specs  &  standards  as  element  of  acquisition 
practices  and  toolset 

*  Define  technical  practices  and  expectations  by 
government 

•  Define  the  “what”  -  and  not  the  “how  to” 

*  Establish  “Select”  list  of  space  systems  standards 

*  Establish  baseline  set  of  common  specs  and  standards 

*  Include  military  and  industry  (e.g.,  AIAA,  ISO)  standards 

*  Establish  Organizational  Policy 

*  Specify  critical  standards  in  RFP 

*  Compliance  Documents 

*  Baseline  contractually 

THE  AEROSPACE 


CORPORATION 


•  Revised  SMC  S&S  List 
published  8/9/2006 

•  65  essential 
documents 

•  Entire  SMC 
System  Portfolio 

•  Military, 
International,  and 
Industry  Standards, 
and  Aerospace 
TORs 

•  Updated  standards 
to  reflect  current 
best  practices 

•  Additional  updates 
to  current 
document  versions 
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SMC  S&S  List 

■  ■■■■■■III 


Qualification  and  Quality  Requirements 
for  Space  Solar  Cells 


THE  AEROSPACE 
CORPORATION 


Standards  Technical/Functional  Areas 


Program  Management 
Systems  Engineering 
Risk  Management 
Configuration  Management 
Design  Reviews 
Product  Assurance 
Electrical  Power 
Electrical  Power,  Batteries 
Electrical  Power,  Solar 
EMI  /  EMC 

Environmental  Engineering 
Human  Factors 
Interoperability 
Logistics 

Parts  Management/Engr 


Ordnance 

Pressure  Vessels 

Reliability 

Maintainability 

Manufacturing  / 

Producibility 

Mass  Properties 

Safety 

Security 

Software  Development 

Structures 

Survivability 

Moving  Mechanical 
Assemblies  (MMAs) 

Test,  Ground 

Test,  Space 

THE  AEROSPACE 


CORPORATION 


•  Issued  by  Lt.Gen.  Hamel  11  July 

•  Establishes  specifications  and 
standards  as  an  integral  element  of 
SMC  acquisition  processes 

•  Applies  to  all  new  development, 
acquisition  and  sustainment 
contracts,  including  new  contracts 
for  legacy  programs 

•  Contractual  compliance  through 
the  supplier  chain,  as  appropriate 

•  SMC  Chief  Engineer  (CE) 
responsible  for  master  list  of 
compliance  documents 

•  SPO’s,  with  CE,  generate  tailored 
set  of  specs  and  standards  and 
recommend  to  PEO  for 
implementation 

•  SMC/CC/AFPEO  -  Space  resolves 
issues 
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DEPARTMENT  OF  THE  AIR  FORCE 

headquarters  space  and  missile  SYSTEMS  CENTER  (AFSPC) 

LOS  ANGELES,  CA 


1  1 

MEMORANDUM  FOR  SMC-ALL 
FROM:  SMC/CC 

SUBJECT:  Initial  Policy  on  Specifications  and  Standards  Usage  at  SMC 

l  Thif.policy  est*blish“  use  Of  specifications  and  standards  as  an  integral  element  of  CMr- 
acquisition  processes.  Programs  executed  bv  SMfVA  KPT?r»  c  ,  ^  fSMC 

LIf '  SMa  Ch'ef  En*jncer  shal>  <*  responsible  for  defining,  coordinating  mainiaininn 
updating  and  reporting  the  master  list  of  compliance  document  The  lisUnclud^  d  e  !,t 

programs.  For  existing  programs  and  contracts,  the  SPO’s,  with  the  SMC  Chief  FifoSLr  11 

specifications,  standards  or  implementation  that  arise  between  SMC/EA  and  SPD\  will  lv^ 
brought  forward  to  SMC/CC/AFPEO-Space  for  resolution.  ** 

policy*  ChierEneinCCr  ,ha"  PrePare  “  SMC  OI  to  institutionalize  the  practice  and  intent  of  this 


MICHAEL  A.  HAMEL 
Lieutenant  General,  USAF 
Commander 


I  THE  AEROSPACE 

CORPORATION 


Recent  Actions 


AjCQUISmnM 
TEEHNOLOGT 
4Nn  LDCrSTICS 


OFFICE  OF  THE  UN  DER  SECRETARY  OF  DEFENSE 

30OODETFN£E  PENTAGON 
WASHINGTON,  DC  ^0301-3000 

March  20;  2005 


MEMORANDUM  FOR  I  I  IP.  STANDARDIZATION  EXECUTIVES  OF  TTTT  MILITARY 
DEPARTMENTS  AMU  DEFENSE  AGENCIES 

SUBJECT:  Policy  Memo  05-3,  ‘TUi  ruination  of  Waivers  re  Cite  Mill | try  Specifications  and 
Standards  in  Sii-liciuniurH  and  Contracts’1, 


On  OtLdher  14,  2004,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  uml 
lavishes  signed  the  Defense  AtiniisiliuJi  Guidance.  Paragraph  I  1 .6  uf  urn  Guidance  states  that 
’ic  is  no  longer  required  l;>  obtain  a  waiver  from  (he  Milestone  Dee  is  ion  Authority  re  die  mi  I  il  ary 
sped  fiend  L>n>*  und  standards,  in  solicit  ai  inns  arid  contracts  " 

Wtiiirc  in  I  he  process  of  preparing  u  formal  change  Lu  DoD  4120.24-M,  “Defence 
SiandardL^ation  Program  Policies  and  Procedures,”  to  eliminate  the  waiver  requirement,  horn  Lida 
document  to  h.-.  cun sis lent  with  t.ic  Under  Scerei wry's  direction.  UntiL  sud:  a  forma!  change  can 
tic  issued  hy  the  DoD  Directives  Office,  til  is  po  I  icy  me  mo  rand  ti  m  deletes  Section  C3.B  and  aJJ  of 
iis  paragraphs  and  subparagraphs  jegardine  waivers  f rum  D;sD  4120.24  M. 

I  rcqucbL  IhiiL  you  tcc-ie  appropriate  a.eti nn  in  ensure  that  everyone  in  your  acquisition  and 
Logistics  cum rmmili eh  is  itwaie  that  a  waiver  to  cite  mi lii ary  ispeci  Hendons  and  standards  in 
Kdlidmlions  and  contracts  is  no  lunger  inquired.  As  noted  in  the  Defense  Acquisition  Gui dunce, 
however,  this  waiver  elimination  should  not  he  interpreted  uk  returning  to  the  “old  way  efitoing 
busings”  hut  iis  iccognition  of  ihe  lmIiutuI  change  that  toot  place  in  DoD  regarding  hie  proper 
application  of  specified  I  ion*.  and  Mandanis.  We  need  to  ensure  ihal  Lhust  in  the  acquisition  und 
Logistics  com  muni  lies  have  die  flexibility  lo  program  requirements,  make  good  decisions, 
and  where  appropriate,  require  conformance  to  mill  I  wry  hpac  ill  cations  and  .■mtudards, 

Tf  ihne  are  any  questions  about  diis  policy  memo  rand  ulh  or  the  slut  us  of  i  lie  change  lo 
DoD  4120.24-M,  my  point  uf  contact  Is  Mr.  Stephen  I  .owe  1 1  at  (70  J>  7G7  G&7  9  Of  email 
Stephen  1 1  'well  ijMta.nsiJ. 


Louis  A.  Kratz 

Assistant  Deputy  IlnLltrSecieLai'y  of  Defense 
(Logistic*  PI  a  n.i  and  Programs) 


OUSD/AT&L  Policy  Memo 
05-3,  “Elimination  of 
Waivers  to  Cite  Military 
Specifications  and 
Standards  in  Solicitations 
and  Contracts” 


Translation:  “You  are  now 
free  to  use  the  right  tool 
for  the  job!” 


UUirAIIK  F©K(sB 


ASC  SE  Road  Show  -  2005 

Rapidly  delivering  war-winning  capability 


•  Overview  -  Gary  Van  Oss 

•  SE  Tool  Set 

-  SE  Toolset  Foundation  -  Charles  Gebhard 

-  Modeling  and  Simulation  -  Pat  Montanaro 

-  Product  Integrity  -  Bill  Kinzig,  Rich  Stepler 

—  Break  — 

-  Airworthiness  -  Bob  Fitzharris 

-  Software  -  Mike  Nicol 

-  Environmental  -  Alex  Briskin 

i=>  -  Specs  and  Standards  -  Scott  Kuhnen 

•  SE  Plans  -  Gary  Van  Oss 

•  Wrap  up  /  Q  &  A  -  Gary  Van  Oss 


Defense  Standardization  Program 

ASC/EN  is  responsible  for  development  and 
maintenance  of  Engineering  Standards  under 
Defense  Standardization  Program  (DSP) 

-  Mandated  by  Public  Law  82-436;  DoD  5000. 1&2;  DoDD 
4120.24;  DoD  4120.3-M;  AFPD  60-1;  AFI  60-101 

Wing  engineering  tailors  and  applies  standards 

-  Responsible  for  application  feedback  to 
ASC/EN 

Industry  design  teams  also  use  MIL  specs  and 
standards 


It’s  part  of  your  day  job! 


Defense  Specifications 
Defense  Standards 
Qualified  Products  Lists 
Non-Gov’t  Standards 
Int’l  Standards 
etc.  * 


1 


Systems  Engineering 

“Engine” 


PROCESS  INPUT 

*  CuitauLer  Needs, Objective* 
Requiremeuts 

Missions 

«  Measures  of  Effectiveness 
*■  Environment: 

11  Con:traints 

■  Technology  Ease 

•  Prior  Outputs 

^  *  Frogram  Decision 

^  Requirements 

Z  *  Requirements  From 

^  Tailored  Specifications  / 

•> i  and  Standards-  { 


— 

- 

" 


Require  me  d  rs  A  n  alvMS 
Analyze  Mission:  &  Environments 
Identify  Functional  Requirements 
Define  Refine  Performance  A 
De:ign  C  onstraint  Requirements 


Requirements  Loop 

F unc  rionjil  Analy  sis/Allo  cation 
Decompose  to  Lower-Level  Functions 
Allocate  Performance  &.  Other  Ltiniting 
Requirements  To  .AH  Functional  Levels 
DefiEE  Refine  F  jLcfioE.il  Interface:  ilnteinal  Eitemnl:  | 
DefiEE  Refine  Integrate  Functional  Architecture 

Design  Loop 


select  Preferred 
Alternative:, 

Trade-Off  Studies 
Effectiveness  Analyses 
Risk  Management 
C  oufigurafion  Management 
Interface  Management 
Data  Management 
Perfoimnnce-Based 
Progress  MEa.sureLn.eEt 
■■  SEMS 
■■  TPM 

»  Technical  Reviews 


Synthesis 

Transform  Architectures  iTuEtrionai  to  Physical) 
Define  Alternative  System  C  uncepts, 

CoEfigurarion  Items  &  System  Elements 
■  Define  Refine  Physical  Interface:  (Internal.EitErual' 
■  ■  Define  Alternative  Product  &  Process  Solution; 


Feedback  Loop 


PROCESS  OUTPUT 

■  Decision  Data  Ba:e 

11  Deri:ion  Support  Data 
■■  System  Functional  &  Physical 

ArchitEctnrES 

■■  Specifications  &  Baseline: 

■  Balanced  System  Snlntin us 


i/m/  HM^aiS'iin 


My  Assertion... 

•  Specs  &  Standards  are  not  gone! 

-  We  are  “down  to”  only  12,000  in  the  aero  sector 

•  Spec  &  Standards,  and  all  the  work  it  takes  to 
create  them,  coordinate  them,  update  them, 
understand  them,  use  them,  is  “foundational”  to  the 

execution  of  the  SE  process  (not  a  “crutch!”) 

•  Development  of,  use  of,  translation  of  technical 
requirements  is  the  heart  of  the  technical  portion  of 

the  SE  process . as  we  revitalize  SE,  consider  the 

role  that  specifications  and  standards  play  in  the 
overall  “business”  of  systems  engineering. 


Benefits  of  the  DSP 


Standards  are  “foundational”  to  all  that  we  do 

-  Measuring  program  execution,  success  and/or  failure 

-  Moving  both  the  State-of-the-Art  and  managing  the 
T  ried-and-the-T  rue 

-  Reducing  risks  in  programs  and  in  the  SE  process 

-  Providing  “confidence”  to  those  who  actually  execute 
the  SE  process 

-  Documenting  &  Communicating  Lessons  Learned 

-  “Mentoring”  the  Next  Generation  (“Here  kid,  read  this!”) 

-  Communicating  technologies  and  strategies  across 
entire  sectors... forming  a  common  understanding 

-  ..Shall  I  continue...? 


AFRL  Contracted  Study 


AFRL  RX  (formerly  ML)  contracted  an 
analysis  of  their  specs  &  stds  workload  in 
2008-09. 

Draft  report  in  works... (excerpt): 

“Military  specifications  and  standards  served  as: 

a.  Procurement  documents. 

b.  A  record  of  experiences  and  lessons  learned. 

c.  Benchmarks  in  system  acquisition. 

d.  A  resource  for  subject  matter  experts. 

e.  A  tool  for  mentoring  and  transferring  knowledge. 

f.  An  aid  in  developing  and  transitioning  new  R  &  D  programs  and 
transitioning  technology.  “ 


HQ  AFMC/ENS  Stdzn  Study 

•  Request  of  AFMC  Centers  for  "key"  specifications, 
standards,  or  handbooks  which  they  would  like  to 
see  returned  to  their  SE  Toolset? 

•  Initial/Raw  Results:  104  different  documents 
requested  for  possible  re-instatement. 

•Approximately  25  of  these  were  requested  by 
multiple  organizations. ..covering  such  topics  as: 
Reliability,  SE,  Config  Mgt,  Corrosion,  Software, 
Materials,  Reviews  &  Audits,  FMECA...many  being 
related  to  what  we  call  standard  practices. 


What  can  you  expect  from  AF? 


AFMC  D&SWS  Council  has  endorsed  a  continued 
study  of  reinstating  and  using  key  stdzn  docs. 

-  Planning  timeline  due  in  June  of  2010 

SAF/AQRE  appreciates  that  certain  "standard 
practices"  (rather  than  "guides")  would  be  useful 
to  restoring  both  common  understanding  and 
discipline  back  into  acquisition. 

Industry  involvement  is  critical. ..again  this  time! 

We  solicit  your  interest  &  support 

-  The  stars  may  be  aligning.. .again... 


No,  seriously... 


Contact 


Mr  Robert  B.  Kuhnen 
HQ  AFMC/ENS 
robert.kuhnen(5)wpafb.af.mil 


Back  Ups 


Standards  Live! 

DoD  Document  Summary  (Active  &  Inactive) 


-  Specifications 

18,834 

-  Standards 

864 

-  Handbooks 

406 

-  CIDs 

4,941 

-  DIDs 

1,019 

-  QPL’s 

758 

-  Non  Gov’t  Standards 

9,223 

-  International  Standards 

1,961 

..in  all  the  Services/Agencies 


Preparing  Activity  by  Service  (Active  &  Inactive) 


-  Air  Force 


2,610 


-  Army 


8,891 


-  Navy 


10,321 


-  DLA 


13,908 


Which  Standards  Matter  to  ASC? 


Def.  Stdzn  documents: 

Military 

NGS 

Total 

Preparing  Activity 

371 

363 

734 

(speaks  for  DoD) 

AF  Custodian 

6356 

2742 

9098 

(speaks  for  AF) 

AF  Review  Activity 

1140 

265 

1405 

(reviews  for  ASC) 

7867 

3370 

11,207 

•  Design  Handbooks  (17) 

-  Shipping  only  1-  and  2-series  documents  today  -  on  CD 

•  AF  Characteristics  Guides  (6) 

-  Shipping  only  -  have  only  begun  migration  to  CD 

•  Misc.  support  to  other  technical  docs  &  publications 

•  Bottom  Line:  Each  of  the  sectors  (Space,  Aeronautical 
Maritime)... all  have  a  body  of  knowledge... standards. 
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Introduction 


NPS  tasked  by  the  Army  Simulation  Proponent 
Division  to  develop,  deliver,  and  sustain  an 
executive  level  course 


Course  sponsored  by  the  Simulation  Proponent 
Division,  office  of  the  Army  Modeling  & 
Simulation  Directorate  for  the  US  Army 


Course  Developed  &  Delivered  by 
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Army  Advanced  SIM  Course 


Course  Purpose: 


Two  week  senior  leaders  course  for  LTC/COL 
level  FA57  and  senior  civilians  in  the  Army 
M&S  Community. 

Focus: 

Provide  a  NON-technical  perspective  of 

^  * '  i  lmn 

significant  M&S  issues. 


NAVAL 

POSTGRADUATE 

SCHOOL 


Army  Advanced  SIM  Course 


Scope: 

•  Equip  students  with  better  M&S  management 
skills  at  Direct  Reporting  Unit  (DRU),  Army 
Command  (ACOM),  and  Program  Executive 
Officer  (PEO)  level. 

•  Familiarize  students  with  current  M&S 
management  concerns  throughout  the 
Acquisition  Life  Cycle,  specifically  the 
different  M&S  applications  in  use  during  each 
phase. 
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Army  Advanced  SIM  Course 


Prototype  Course: 


•  Objective:  Discover  educational  gaps,  identify 
appropriate  target  audience,  and  determine 
potential  for  future  offerings. 

|  •  Delivered:  11-22  May  2009 

•  Location:  NPS  Center  for  Executive  Education  (CEE) 
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Army  Advanced  SIM  Course 


Prototype  Course: 


Student  Body:  20  Total 


Active 

Duty 

Officer 

70% 
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Army  Advanced  SIM  Course 


*  -  -  ™ 


m I,! 

k,«P 

S'  l| 

P  iiliil 

L  $ 


JMSC  DITRA 

5%.  5% 


Organizations  Represented 


ATEC 

10%_ 


PEO-STRI 

5% 

ATC-OTC 
5% 

FFID 

5% 

TRADOC- 
ARCIC  Nat  l  SIM  Ctr. 
5%  10% 


MSCO 
5% 

2/75  TSD 
5% 

EUCOM 
5% 

_HQDA, 

Army  G 3 
15% 
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Army  Advanced  SIM  Course 


Prototype  Course:  NPS  Department  Participation 


-  Computer  Science 

-  Graduate  School  of  Business  &  Public  Policy 

-  Modeling,  Virtual  Environments,  and  Simulation 
(MOVES) 

-  Operations  Research  &  Simulation  Experiments  & 
Efficient  Designs  (SEED)  Center 

-  Systems  Engineering 
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Army  Advanced  SIM  Course 


Prototype  Course:  SME  Participation 

-  NPS  Thesis  Students 


-  GSResearch  LLC 
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Army  Advanced  SIM  Course 


Prototype  Course:  Breakdown 


•  Module  1:  Management  Concerns  Regarding  M&S 

-  Week  1  (Lessons  1-5) 

•  Module  2:  Lifecycle  M&S  Issues 

-  Week  2  (Lessons  1  -4,  plus  Guest  Lectures) 


rH 
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Army  Advanced  SIM  Course 


Prototype  Course:  Topics 

Module  1:  Management  Concerns  Regarding  M&S 

•  M&S  Education  (Lesson  1) 

-  Role  of  SIM  professional,  development  &  significance  of  MSBOK,  and  what  should  an 
M&S  curriculum  contain  for  DoD  M&S  professionals 

•  M&S  Requirements  (Lesson  2) 

-  Overview  of  DoD  M&S,  M&S  Standards,  LVC,  M&S  Requirements  Generation,  M&S 
Fidelity  and  Resolution,  VVA,  DoD  Stakeholders,  DoD  Organizations,  Governing  Docs, 
Army  M&S  Domains 

•  M&S  in  Testing  (Lesson  3  &  4) 

-  T&E  Overview,  Why  We  Test,  Different  Testing  Types  (DTE,  OTE,  LFTE  and  Military 
Experimentation) 

•  M&S  in  Analysis  (Lesson  5) 

-  Overview  of  DoD  Analysis  Domain  and  Development  of  Analytical  Simulation  Study 
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Army  Advanced  SIM  Course 


Prototype  Course:  Topics 


Module  2:  Lifecycle  M&S  Issues 

•  M&S  Acquisition  (Lesson  1  &  2) 

-  Focus  of  M&S  in  Acquisition,  JCIDS  and  use  of  M&S  in  JCIDS,  DoD  Life  Cycle  and  M&S 
in  DoD  Life  Cycle,  M&S  Challenges  in  Acquisition  over  the  DoD  Life  Cycle,  M&S 
Contracting  Considerations,  SIM  Based  Acquisition  (SBA) 

•  M&S  in  Risk,  Cost,  and  Decision  Analysis  (Lesson  3) 

-  M&S  in  Risk  and  Cost  Analysis  in  Program  Management,  M&S  support  in  Decision  Making 

•  Future  Trends  in  Simulation  (Lesson  4) 

-  M&S  Convergence  of  Live,  Virtual,  and  Constructive  Simulation 

-  Serious  Games  and  massively  Multiplayer  Games,  Agent  Based  Models,  Other  New  M&S 
Trends 

•  Guest  Lectures  (Provided  expert  knowledge  and  techniques) 

-  Concept  Refinement 

-  M&S  Education  Opportunities  for  DoD  Communities 
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Army  Advanced  SIM  Course 


Prototype  Course:  Supplemental  Materials 

•  Case  Studies 

•  Lesson  Pre-Reads 

•  Handouts 

•  Group  Exercises 

•  In-Class  Tool  Application 

•  Instructor  Lead  Discussion 


WWW.NPS.EDU 
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Army  Advanced  SIM  Course 


Lessons  Learned... 
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Army  Advanced  SIM  Course 


•  Course  Assessments 

-  Each  student  completed  a  daily  assessment,  a  weekly 
evaluation,  and  overall  course  evaluation. 

-  Each  daily  evaluation  focused  on  the  topic  delivered  that 
day 

-  Student  feedback  will  be  incorporated  into  future  course 
offerings 

•  Some  Changes: 

-  FA57s  Only 

-  Incorporate  more  SMEs  from  DoD  and  Industry 

-  More  Practical  Exercises 


WWW.NPS.EDU 
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Army  Advanced  SIM  Course 


The  Way  Forward... 


imw 
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Army  Advanced  SIM  Course 


Next  Course: 


26  April  -  7  May  2010 


Naval  Postgraduate  School  -CEE 


/  i 
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Army  Advanced  SIM  Course 


Contact  us  for  more  information... 


Naval  Postgraduate  School  -SE  Department 


llllll 


*  FI 
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Dr.  Gene  Paulo 

Course  Facilitator  &  Instructor 
(831)656-3452 

eppaulo@nps .  edu 

Ms.  Stephanie  Few 

Research  Assistant 
(704)274-5178 

smfew@nps.edu 
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How  the  Army  Uses  M&S 


Analysis 

Community 


Conduct  Ana  lyses  on  performance, 
effectiveness,  survivability,  tradeoffs, 
cost/benefit,  logistical  systems, 
personnel  systems ,  army  force 
structureand  risk. 


•  Combat  XXI, 

•  AWARS 

•  CASTFOREM 

•  Janus 

•  Spreadsheet  Modeling 

•  CMS2 

•  APHAKS 

•  JICOM 

•  AMP 

•  MIDAS  and  GAMS 

Total  Army  Analysis  products, 
determining  effectiveness  of  weapon 
systems,  determining  performance  and 
system  characteristics  of  equipment 
and  system  purchases,  determining 
effectiveness  of  force  management 
decisions 

CAA,  TRAC,  AMSAA 


Provide  actionable  events  designed  to 
stimulate/simulate  the  proper  HUMINT, 
SIGINT,  and  GEOI  NT  intelligence 
disciplines  gathered  in  real  world 
operations. 


•  TACSIM 

•  BSS 

•  IEWTPT 

•  CSP 

•  MUSE 

•  NWARS 

•  QSIDS 

•  JCATS 

•  JLCCTC  Federation 


Training  intelligence  personnel  for  COIN 
operations,  and O I F/OEF activities 


ArmyG2,  USAIC,  JICTC, 
Intelligence  soldiers  in 
the  Operational  Army 


Planning 

Community 


Conduct  a  system  of  systems  approach 
to  various  planning  tools  (i.e.  Force 
Management  Model 


•  ARFORGEN 

•  CFAST 

•  APIT 

•  APEX 

•  MIDAS 

•  EUST 

•  APOD/AST 

•  INFO-21 

•  JOPESand  JFAST 

•  JDLM 

•  ADEPT 

Determ  i  n  i  ng  wa  rfighti  ng  ca  pa  b  i  I  ities 
req  u  i  rements,  co nd  ucti  ng  resea  rch  a  nd 
development,  and  providing  resources. 
ARFORGEN  analysis  and  modeling. 


HQDA,  Army  Command, 
ASCCs,  DRUs,  Operational 
Units  Staffs 
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How  the  Army  Uses  M&S 


Experimentation 

Community 


Exploring,  testing  and  validating 
warfighting  ideas  and  concepts  to 
transform  the  way  soldiers  fight  future 
battles . 

•  JLCCTC  Federation 

•  HITLsims  and  simulators 

•  Controlled  field 
experiments 

•  BLCSE 

•  M  ATREX  a  nd  3CE 

•  ATIN 

•  OTC 

•  ACRT 

•  AWARS 

•  OneSAF 

•  EDSIM 
-  CMS2 

•  FireSIM 

•  CAMEX 

Conduct  of  experiments  involving 
soldiers  and  leaders  within  the  live, 
virtual  and  constructive  environments 
fo  r  exp  lo  ri  ng  co  ncepts,  ca  pa  b  i  I  ities 
requirements  and  solutions  across 
DOTMLPF  and  the  AC2DP  process 

TRADOC,  RDECOM,  SMDC,  USASOC, 
AMEDD,  PEOs,  Battle  Labs,  FORSCOM 


Acquisition 

Community 


Provide  virtual  and  constructive  test  beds 
thru  which  weapon,  equipment  and 
ammunition  factors  can  be  prototyped, 
tested  and  evaluated  during  the  acquisition 
process. 

•  ANSYS  LS-DYNA 

•  ProE,  ANSYS 

•  OneSAF 

•  MSC  Adams 

•  MPR3D 

•  AJEM 

•  MUVES 

•  CB  Sim  Suite 

•  ALOHA 

•  PVTM 

•  Visual  Weight 

•  MUVES 

•  SIV 

•  Casualty  Reduction  Model 

•  BlastX 

Reduces  testing  time  and  costs  and  allows 
measurement  of  phenomenon  that  can  not 
be  measured  using  traditional  methods. 
Provides  data  for  procurement  decisions. 
Allows  for  selection  and  characterization  of 
optimal  material  solutions. 

ASAALT,  RDECOM,  PEOs 


Testing 

Community 


Employ  M&S  throughout  program  life 
cycle  to  support  requirements  definition; 
design  and  engineering;  test  planning, 
rehearsal, and  conduct  of  an  Army  test. 

•  Live/ Virtu  a  I/Constructive 

•  4DWX 

•  Overarching 
Contamination  Avoidance 
Model 

•  CBSNE 

•  DSOM 

•  JLCCTC  Federation 

•  ECSM 

•  IMASE 

•  COLPRO 

•  DETES 

•  DMS3 

•  EGI/CEGS 

•  ETS 

•  NETS 

Evaluate  the  performance  of  tested 
items ,  systems  a  nd/or  organizations, 
early  examination  of  soldier  interface 
and  missions,  determines  system 
perfo rma nee  and  safety. 


ArmyT&E,  ATEC 
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How  t 
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What 


Impact 


Who 

_ / 


Training  & 
Operational 
Community 


Delivering  an  integrated  Live,  Virtual 
a  nd  Co  nstructi  ve  tra  i  n  i  ng  en  vi  ro  n  ment 
that  support  the  ARFORGEN  model  and 
mission  rehearsal  requirements. 


•  JLCCTC  Federation 

•  Virtual  Simulators 

•  CCTT 

•  AVCATT 

•  EST  2000 

•  OneSAF 

•  JNEM 

•  DCARS 

•  WARSIM 

•  IEWTPT 

•  jrvnc 

•  LVC-IA 

•  Gaming 

Pre-deployment  training  exercises, 
mission  rehearsals,  trained  and  ready 
soldiers,  ABCS  &  digital  system  training 


CTCs.  NTC,  JFCOM, 
BCTCs,  all  Army  units 


WWW.  N  PS  .BDt  f 


Army  Uses  M&S 


Logistical 

Community 


M&S  develops  a  level  of  understanding  of 
the  interaction  of  the  parts  of  the  logistical 
system,  and  of  the  logistical  system  as  a 
whole,  seldom  achievable  via  any  other 
process. 


•  JDLM 

•  MIDAS 

•  JFAST 

•  EUST 

•  TARGET 

•  PORTSIM 

•  POPS 

•  AMP 

•  JCATS 

•  JLCCTC  Federation 

•  LOGFED 


Deployment  timelines,  training  soldiers 
in  logistical  functions  and  units, 
efficiencies  of  logistical  operations 


AMC  Logistics  Support  Activity , 
SDDCTEA,  soldiers  in  logistical  units 
and  staffs,  AMC,  NSC,  CASCOM 
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Surfboard  Diagram 


New  M&S  Governance  and  Management  Structure  Organized  by  Communities. 

Designed  to  Support  &  Integrate  M&S  Activities  across  the  Department. 

Led  by  a  1  to  2  Star  M&S  Steering  Committee  (M&S  SC)  to  provide  governance. 


Analysis 

PA&E 
&  JS 


Planning 

JS 

&  Policy 
/  \ 


Testing 

DOT&E 

&AT&L 


I 


I 


Experimentation 

JFCOM 


Common  and  Cross-Cutting  M&S  Tools 


Common  and  Cross-Cutting  M&S  Data 


Common  and  Cross-Cutting  M&S  Services 


Components 

OSD,  Joint  Staff,  COCOMs,  Services 


Goal:  Establish  corporate  M&S  management  to  address  DoD  goals: 
Leads/guides/shepherds  the  $Bs  in  DoD  M&S  investments;  adds  value  thru 
metrics  &  ROI-driven  priorities;  and  seeks  to  provide  transparency. 


Student  Daily  Assessment  Results 
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Advance  Simulation  Course:  Daily  Assessment 

Presenters): 

Session  Title: 

Date: 


Please  rate  each  item  on  the  1  to  5  scales,  where  1  is  the  lowest  rating  and  5  is  the  highest 

Two  different  topics  are  delivered  by  different  instructors  on  some  of  the  course  dates;  therefore,  some 
questions  are  repeated  to  access  each  topic  separately.  Topic  one  represents  either  the  morning 
session  or  one  full  day  of  instruction.  When  applicable,  topic  two  represents  the  afternoon  session. 

1.  The  objectives  were  dearly  explained:  1  2  3  4  5 

2.  The  content  was  frequently  reinforced  with  examples: 

3.  The  content  was  relevant/useful  to  my  current  or  future  job: 

4.  The  va  lue  of  the  lecture{s): 

5.  The  value  of  the  class  discussion: 

6.  The  instructor  was  well  prepared  for  topic  one: 

7.  The  instructor  was  well  prepared  for  topic  two  (if  applicable): 

8.  The  concepts  and  ideas  were  clearly  presented  for  topic  one: 

9.  The  concepts  and  ideas  were  clearly  presented  for  topic  two  (if  applicable): 

10.  The  instructor  focused  on  the  applications  of  concepts  for  topic  one: 

11.  The  instructor  focused  on  the  applications  of  concepts  for  topic  two  (if  applicable): 

12.  The  instructor  interacted  effectively  with  participants  for  topic  one: 

13.  The  instructor  interacted  effectively  with  participants  for  topic  two  (if  applicable): 

14.  Overall  rating  for  topic  one: 

15.  Overall  rating  for  topic  two  (if  applicable): 

16.  The  value  of  pre-reads  and  handouts  for  topic  one  (leave  blank  if  n/a): 

17.  The  value  of  pre-reads  and  handouts  for  topic  two  (leave  blank  if  n/a): 

18.  The  value  of  the  case  studies  for  topic  one  (leave  blank  if  n/a): 

19.  The  value  of  the  case  studies  for  topic  two  (leave  blank  if  n/a): 

20.  The  value  of  small  group  exercises  for  topic  one  (leave  blank  if  n/a): 

21.  The  value  of  small  group  exercises  for  topic  two  (leave  blank  if  n/a): 

22.  The  overall  rating  for  topic  one: 

23.  The  overall  rating  for  topic  two  (if  applicable): 

Comments: 


20  Assessments  Completed  Per  Day 

Totaled  the  #  of  ratings  (1-5)  per 
assessment,  added  all  assessment 
ratings  together  per  day  for  an  overall 
idea  of  the  day’s  topic  likes,  dislikes, 
and  delivery. 

Each  rating  (1-5)  total  was  divided  by  # 
of  questions  completed  per  day  (some 
questions  did  not  apply)  to  reach  the 
percentages. 


Comment  Space  provided  to  list 
3  Likes  and  3  Dislikes  of  the  day. 


1  -Lowest,  5  -Highest 
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Army  Advanced  SIM  Course 


NFS  &  SME  Recognition 

Curtis  Blais:  NPS  CS  &  MOVES 


Jeff  Cuskey:  NPS  GSBPP 
Chris  Darken:  NPS  CS  &  MOVES 


John  Dillard:  NPS  GSBPP 

Karl  Gunzelman:  GSResearch  LLC 

Tom  Hoivik:  NPS  OR 

Mathias  Kolsch:  NPS  CS  &  MOVES 

Tom  Lucas:  NPS  OR  &  SEED  Center 

Dave  Matthews:  NPS  GSBPP 

Don  McGregor:  NPS  CS  &  MOVES 

David  Olwell:  NPS  SE 

Gene  Paulo:  NPS  SE 

Mark  Rhoades:  NPS  SE 

Susan  Sanchez:  NPS  OR  &  SEED  Center 
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Naval  Postgraduate  School 
Advanced  Seabase  Enabler  Project: 
A  Systems  Engineering  Case  Study 


Lance  Flitter 

Naval  Surface  Warfare  Center 
Carderock,  MD 


Gene  Paulo 
Associate  Professor 
Department  of  Systems  Engineering 
Naval  Postgraduate  School 


Paper  8880 


October  29, 2009 


Project  Overview 


■  This  project  describes  a  NPS  capstone  project  as  part  of  obtaining 
an  MS  in  Systems  Engineering. 

■  The  project  examined  transporting  cargo  from  a  sea  base  to  the 
desired  destination  and  make  recommendations  regarding  the  best 
approaches  for  meeting  those  objectives.  Key  research  goals  were: 

>  Determine  required  capabilities  and  functions  for  an  ASE 

>  Develop  appropriate  operational  concept  (OPCON) 

>  Examine  various  ASE  concepts,  to  include  the  Transformable 
Craft  (T-Craft) 

v  Conceptual  platform  under  the  design  of  the  Office  of  Naval 
Research  (ONR) 

v  ONR  sponsoring  multi-year  effort  through  NPS  to  assist  in 
T-Craft  system  design  and  architecturing. 
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ASE  Project  Stakeholders 


Major  Stakeholders: 

■  Operational  Commands 

■  Navy 

■  USMC 

■  Army 

■  SOCOM 

■  JFCOM 

■  TRANSCOM 


Secondary  Stakeholders: 

•k  NAVSEA 

■  Marine  Corps  Combat 
Development  Command 
(MCCDC) 

■  NAVSUP 

■  Marine  Corps  Logistics 
Command 

■  ONR 

■  TACOM 

■  Combined  Arms  Support 
Command  (CASCOM) 

■  Military  Sea  Lift  Command 

■  PDM  Army  Watercraft  Systems 
(AWS) 

■  NATO 

Coast  Guard 


Systems  Engineering 
Development  Process 


Formulation 

▲ 


Analysis 

♦ 


Interpretation 


Primary  Data  Flow 
+  Secondary  Data  Flow 


Final  Problem  Statement 


“A  capability  is  needed  to  fully  enable  the  potential  of 
the  sea  base.  For  a  sea  base  to  be  truly  beneficial  a 
capability  must  exist  that  supports  efficiently 
transporting  needed  materiel  from  the  sea  base  to  the 
desired  debarkation  point.  The  capability  must  support 
peace-time,  non-combat  operations’  and  war-time, 
combat  operations’  logistics  and  support  needs.  The 
solution  must  be  cost  effective  and  capable  of  operating 
under  all  environmental  conditions,  including  sea  states, 
under  which  necessary  military  operations  are  expected 
to  take  place  and  must  support  a  transport  rate  sufficient 
to  ensure  materiel  is  delivered  within  operational  time 
requirements.” 
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Frequency 
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Seabasing 

Analysis  of  the  Full  Range  of  Military  Requirements 
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System 


1.0 

Deploy  ASE 

(This  function  moves  the 
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ASE  Objectives  Hierarchy 
Non -Functional  Requirements 


ASE  Non-Functional 
Requirements 


Alternatives 


■  Researched  several  possible  alternative  solutions 

■  LCAC  (Landing  Craft  Air  Cushion) 

■  SSC  (Ship  to  Shore  Connector) 

■  LSV  (Logistics  Support  Vessel) 

■  LCU  (Landing  Craft,  Utility) 

■  T-Craft  (Transformable  Craft) 

■  JHSV  (Joint  High  Speed  Vessel) 

■  Heavy  Airlift 

■  All  options  already  conceived.  Information  gathered  from 
existing  projects  /  programs. 


Systems  Analysis 


■  Both  static  and  dynamic  systems  analysis: 

>  Using  cargo  capacity,  speed,  range  of  various  alternatives, 
team  members  conducted  comparison  for  both  MCO  and 
Humanitarian 

>  Simulation  analysis  examined: 

v  Extent  of  capability  to  complete  the  Load  Cargo 
function,  using  Mean  time  required  to  assemble 
cargo/forces  and  load  assembled  cargo/forces. 

•/  Extent  of  capability  to  complete  the  Transport  Cargo 
function,  using  mean  time  required  to  transport 
cargo/forces  ashore. 
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Overall  Results 


■  T-Craft  has  highest  overall  performance  score  by  a 
large  margin  for  combat  mission 

■  JHSV  has  highest  score  for  humanitarian  mission 
followed  by  LSV  and  T-Craft 


?Cost  vs  Performance  - 


Surge  Operation  -Seabase  @  25NM  - 
#  of  vessels  to  move  13k  tons  in  10  hrs 
@  MAX  Speed 


T-Craft  JHSV  SSC  LCAC  LCU  LSV 


T-Craft  offers  the  highest  utility  with  a 
moderate  cost 

LSV  and  LCU  provide  moderate  utility  at 
comparatively  low  cost 

SSC  and  LCAC  costs  are  the  highest  with 
the  lower  utility  value  due  to  their 

relatively  small  cargo  capacity 


Utility  Vali 


jor  Combat  Operations 


Major  Combat  Mission  -  Surge  Operations 
13k  tons  in  10  hours 

Total  Craft  Acquisition  Cost  vs.  Utility  Value 


Acqusition  Cost  of  total  #  connectors  required  for  surge  (FY2009 

$M) 
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P  Cost  vs  Performance  -  Humanitarian  Mission 


Humanitarian  Mission  -Seabase  @  25NM  - 
#  of  vessels  to  move  100k  tons  in  48  hrs 
@  max  craft  speed 


■  JHSV  offers  the  highest  utility  with  a 
moderate  comparative  cost 

■  LCU  and  LSV  again  have  the  lowest 
cost  with  LSV  having  the  lowest  overall 
cost  with  a  fairly  high  utility 

SSC  and  LCACs  are  obviously  poor 
choices  with  high  cost  and  low  utility 
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FY  2010  (and  Beyond)  Proposed  Research 

■  Conduct  thorough  systems  analysis  of  T-Craft  and 
Sea  Base  Enablers 

■  Examine  more  specific  proposed  T-Craft  capabilities 
and  their  operational  impact 

■  Examine  other  operational  concepts  and  scenarios 
appropriate  for  T -Craft  system. 

■  Develop  a  virtual  representation  of  T-Craft  for  use  in 
analysis  and  possibly  training 
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Focus  for  FY  2010 


msBsf 


In  depth,  formal  evaluation  of  T-Craft  performance,  based  on  the 
systems  engineering  framework  and  operational  concept 
developed  as  part  of  F Y 09  effort. 

■  This  includes  the  evaluation  of  alternatives  and  decision 
analysis  using  simulation  and  experimental  design. 

■  Researchers  from  Simulation  and  Efficient  Experimental 
Design  (SEED)  Center  at  NPS  support  this  effort 

■  Actively  recruiting  an  NPS  Operations  Research  student  to 
address  this  topic  for  their  thesis. 


*  * 


Focus  for  FY  2010 


■  Examine  and  determine  the  requirements  for  the  broad 
simulation  analysis  efforts  addressing  T-Craft  performance  in  a 
variety  of  operational  concepts. 

■  Examine  and  determine  the  appropriateness  of  several 
simulation  tools  to  meet  these  requirements  regarding  the 
evaluation  of  T-Craft  performance  in  a  variety  of  operational 
concepts. 


Focus  for  FY  2010  (and  beyond) 


Create  a  prototypical  virtual  environment  (VE)  that  can  be  used 
by  designers  of  the  T-CRAFT  to  inspect  prospective  designs. 

■  For  example,  inside  the  virtual  environment,  the  designers 
will  be  able  to  simulate  various  operations  in  the  craft  to 
ensure  that  the  design  can  support  these  operations. 

■  Additionally,  the  VE  will  be  integrated  into  a  Java-based 
discrete-event  simulation  package  known  as  SimKit.  This 
integration  will  provide  a  means  to  conduct  very  powerful 
simulation  analysis  in  a  virtual  environment.  At  least  one 
thorough  test  case  of  this  integrated  simulation  will  be 
developed  and  demonstrated. 


Focus  for  FY  2010  (and  beyond) 


■  Conduct  a  thorough  life  cycle  cost  analysis  of  the  T-Craft.  I 
have  an  OR  student  who  is  very  interested  in  this  as  his  thesis 
topic. 

■  Propose  and  develop  a  “fleet”  architecture,  examining  different 
combinations  of  cargo  platforms,  to  include  T-Craft,  that  can 
best  address  transportation  gaps  identified  by  Army  and  USMC 
in  specific  detailed  scenarios.  This  effort  will  build  on  our 
previous  systems  architectural  development,  which  considered 
individual  cargo  platforms  in  competition  with  each  other 
(LCAC,  JHSV,  T-Craft,  etc).  This  portion  of  the  project  is 
being  conducted  as  a  graduate  thesis  by  an  NPS  student  in  the 
Master  of  Science  in  Systems  Engineering  curriculum. 


*  * 


Contact  Information 


Questions? 


Gene  Paulo 

Associate  Professor,  Department  of  Systems  Engineering 

Office:  (831)  656-3452 
Email:  eppaulo@nps . edu 
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DRIVE  Developments,  Inc. 


12th  Annual  Systems  Engineering  Conference 

“Achieving  Acquisition  Excellence  via  Effective  Systems  Engineering.” 

San  Diego,  CA 
October  29th,  2009 

“Tactical  Wheeled  Vehicle  Integrated  Diagnostics  - 
The  Past ,  Potential,  Politics,  and  Path  Forward" 

Larry  Osentoski,  President  &  CEO 

DRIVE  Developments,  Inc. 

7314  Nineteen  Mile  Road 
Sterling  Heights,  MI  48314 


DRIVE  Developments,  Inc. 


I Developments,  Inc.H 

Corporate  Overview 

DRIVE  Developments,  Inc.  is  a  high  quality  engineering  solution 
provider.  Our  experience  is  strongly  rooted  in  vehicular  diagnostics  and 
prognostics.  This  is  evidenced  in  our  DIME  (Diagnostic  Information 
Management  Environment)  product  line  which  combines  over  50  years 
of  staff  experience  with  hardware  and  software  development  within  test 
facilities,  remote  diagnostics  and  fleet  management. 

The  breadth  of  our  capabilities  also  extend  into  other  areas  of  vehicular 
engineering  to  include  scientific  test  cell  development,  internet  applications 
and  plant  monitoring  systems. 

‘We  truCy  6e(ieve  in  the  philosophy  of  u Lead,  Follow  or  get  Out  of  the  (Wayn 


DRIVE  Developments,  Inc. 
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Business  Highlights 

•  Michigan  Small  Business  incorporated  in  July  2007 

•  Focused  on  Military  and  Commercial  vehicle  work 

•  Featured  on  The  Economic  Report  with  Greg  Gumbel  in  Fall  08’ 

•  Selected  as  one  of  “Michigan’s  50  Businesses  To  Watch  for  2009” 

•  Selected  as  Prime  on  $500M  /  5  year  TARDEC  OMNIBUS 


DRIVE  Developments,  Inc. 
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‘Tactical  Wheeled  Vehicle  Integrated  Diagnostics  - 
The  Past ,  Potential ,  Politics ,  and  Path  Forward ” 

Overview 


•  PAST  -  Research  to  CRM 

-  Commercial,  Government,  &  Military 

•  POTENTIAL  -  Do  we  really  need  Prognostics  Day  One? 

-  Overview  of  Benefits  from  Diagnostics  to  Prognostics 

•  POLITICS  -  Do  we  need  another  stimulus  package? 

-  Funding  for  development  that  inhibits  deployment 

•  PATH  FORWARD  -  A  bird  in  the  hand? 

-  Success  Stories 


DRIVE  Developments,  Inc. 


The  Past 

•  Commercial  Diagnostic  Deployments 

-  Automotive  OEMS  (Product  development,  CRM,Onstar) 

-  Fleet  Managers  (dispatch,  geofencing,  employee 
monitoring) 

-  Insurance  Carriers  (Tracking,  use,  and  validation) 

•  Government  Research  Pilots 

-  Dept  of  Transportation  (MDOT  DUAP) 

-  1-95  Corridor  activities  (cell  phone  data  for  freeway 
congestion  monitoring) 

•  Military  Research 

-  Congressional  plus  ups  only,  no  requirement  defined  in 
most  cases! 

-  Diagnostics  always  listed  as  objective  requirement,  never 
a  threshold  requirement  in  vehicle  OEM  contracts 
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The  Potential 


Prognostics 


Condition  Based 
Maintenance 


Enhanced 

Diagnostics 


Diagnostics 


DRIVE  Developments,  Inc. 
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The  Potential  (Commercial  Mindset) 

•  Commercial  market  starts  with  diagnostics  on  the  vehicle  platform  to 
diagnose  failures  as  low  hanging  fruit  for  warranty  and  repair 

•  OEMs  gather  data  to  form  baseline  for  Enhanced  Diagnostics  that 
along  the  way  can  provide  CRM  (Customer  Relationship 
Management)  benefit 

•  Seeks  to  develop  prognostic  concepts  that  support  business 
objectives  rather  than  for  the  challenge  or  academic  aspects 


DRIVE  Developments,  Inc. 


Prognostics 


Commercial 

Market 

Perspective 
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The  Potential  (Military  Mindset) 

•  Seeks  prognostics  to  predict  impending 
component  failure  as  first  objective,  heavy 
academic  influence. 

•  Focuses  on  research  to  identify  the  perfect 
prognostic  solution  as  opposed  to  collecting  and 
analyzing  data  on  existing  fleets  and  providing 
immediate  value. 

•  Disparate  systems  have  political  barriers  to 
interoperability,  not  technical  limitations. 


•  Retrofit  more  sensors  onto  vehicles  to  get  more 
data. 


DRIVE  Developments,  Inc. 


Diagnostics 


DRIVE  Developments,  Inc. 
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Immediately  After  the  failure  has  occurred  a 
resident  recording  device  can  gather  data  evidence. 


'Circular  Incident  Buffers 
'Situational  Awareness 
•Location  History 
•Incident  Reporting 
'Application  development : 
•Fuel  efficiency 
•Payload  calculation 
•Weather  Conditions 
•Fleet  Monitoring  Interpolation 


DRIVE  Developments,  Inc. 
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Immediately  After  the  failure  the  resident 
recording  device  forwards  actionable  data  to  depot 


•  Real  Time* 

•Situational  Awareness 
•Location  Tracking 

•Incident  Reporting  KfeiJi  J?* - 

•Application  development : 

•Accident  Detection  /  Reporting 
•Route  Guidance 
•Integration  with  LOGSA 
•Reduced  MTTR  due  to  CBM 
•Increased  MTBF  due  to  CBM,  not  too  early  or 

too  late  based  on  expected  component 
life  and  actual  use. 
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•RIVE 


Developments,  Inc 


Before  the  failure  occurs  the  vehicle  warns  the 
entire  chain  of  command  from  operator  upstream. 

Application  development : 

•Save  lives  through  prediction  of  failures 
•Maximize  Asset  Utilization  through  use  management 
•Advanced  LOGSA  inventory  management  to 
reduce  logistics  tail  be  providing  repair  parts  Just  In  Time 


DRIVE  Developments,  Inc. 


The  Politics 

•  Military  vs.  Commercial  Requirements  (OnStar  like  technology,  you 
can  have  it  as  a  convenience.  Soldiers  can’t  get  it  to  save  their  life.) 

•  Congressional  earmarks  dictate  direction  of  diagnostics  and 
prognostics  for  last  decade!  Mostly  6.1-  6.3  funding  which  is  only 
research  funding.  Rewarding  research  instead  of  rewarding  of 
implementation  working  with  the  customers  in  the  PM  Shops  to  fulfill 
operational  requirements. 

•  Fancy  program  names  create  atmosphere  of  false  prophecy  in 
diagnostics  space.  Past  failures  taint  the  diagnostics  industry. 
“Stenographic,  multifunctional ,  polymer,  language  communications 
system”  -  Rep  Jeff  Flake  (AZ)  describing  how  a  common  ink  pen 
would  be  described  if  included  in  the  Defense  Appropriations  Bill. 

http://www.voutube.com/watch?v=McEz2l1EvDs&feature=channel 

•  PM  Shops  starting  to  fund  their  own  diagnostics  recorder  because 
research  labs  have  been  researching  diagnostics  in  vehicles  for 
over  a  decade  and  have  not  even  developed  or  discovered  the 
recorder  that  should  be  used,  much  less  the  application  of  it.  Some 
areas  are  great  for  research  to  handle,  some  are  not.  Contractors 
have  more  data  than  the  US  Army  on  their  vehicles.  Why? 
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The  Politics,  continued. . . 

•  Not  leveraging  past  experience  of  commercial  markets  because  of 
need  for  continued  research  for  technology  that  already  exists  in  a 
form  that  can  be  utilized  TODAY. 

•  Presently  politics  funding  same  work  over  and  over  (Groundhog  day 
effect)  Same  projects  going  to  same  government  agencies  with 
same  contractors.  Only  change  is  leadership  on  government  side 
enabling  this  game  to  continue,  e.g.  Three  year  rotation  cycle  for 
05’s  and  06’s. 

•  Sexy  language  game  sells  in  congressional  plus  up  world  and 
nowhere  else.  e.g.  Rep  Jeff  Flake  (AZ)  quote  regarding  pen 
description. 


DRIVE  Developments,  Inc. 
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H  The  Path  F orward 


•  Success  Stories 


-  PM  MTV  :  Development  of  Militarized  version  of 
DIME  (Diagnostic  Information  Management 
Environment) 

-FUEL  :  2.6  mpg  savings  on  a  Medium  Tactical 
Vehicle  TODAY!  (6.6  vs.  4mpg) 


DRIVE  Developments,  Inc. 


•  Comparison  of  two  MTVs  in  same  usage  pattern 

•  FUEL  :  6.6mpg  vs.  4mpg  economy  savings 

-  2003-2004  model  year  MTVs  being  driven  by  the  PA  National 
Guard 

-  Vehicles  run  same  routes 

-  Vehicles  are  identical  configurations 

-  Vehicle  A  :  No  issues 

-  Vehicle  B  :  Diagnostic  faults  related  to  fuel  efficiency 
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~w]  View:  Time  (seconds)  ^T| 


Engine  Speed  frpms 


290(H 
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Vehicle  A  (6.6mpg) :  Transmission  state, 
more  detailed  driving  usage 


5,633 


•  34%  of  its  time 
in  neutral 
•31%  of  its  time 
in  7th  gear 

•  45-50%  of  its 
time  at  idle 


30,545  ■  Transmission  Range  - 1  □  Transmission  Range  -  2  ■  Transmission  Range  -  3 

□  Transmission  Range  -  4  □  Transmission  Range  -  5  ■  Transmission  Range  -  6 

□  Transmission  Range  -  7  □  Transmission  Range  -  Nl  □  Transmission  Range  -  R 
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Vehicle  A  :  Basic  Review  of  Healthy  Vehicle 

•  Vehicle  fielded  to  this  guard  unit  spends  approximately  : 

»  34%  of  its  time  in  neutral 
»  31%  of  its  time  in  7th  gear 
»  45-50%  of  its  time  at  idle 

•  This  vehicle  has  numerous  active  fault  codes  some  of  which 
do  not  illuminate  the  malfunction  indicator  lamps. 
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Vehicle  Comparison 

•  When  compared  to  another  vehicle  in  the  same  fleet  the 
presence  of  various  fault  conditions  can  lead  to  varying  data 
and  increased  fuel  consumption. 
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Parameter 

5,000 


Engine  Fuel  Rate 


T]  View:  Time  (seconds)  "7] 


4,000- 


3,000- 


~a 

c. 


ci 
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2,000 


1,000- 


Vehicle  B  (4mpg) :  Fuel  Rate  Data  from 
another  5  year  old  MTV  in  Same  Fleet 


N  j  ifl 

Engine  Fuel  Rate  (Lflhr) 


btHH 
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11,334 


•  29%  of  its 
time  in 
neutral. 

•  33%  of  its 
time  in  7th 
gear. 

•  40-45%  of 
its  time  at 
idle. 


35,578 


■  Transmission  Range  - 1  □  Transmission  Range  -  2  ■  Transmission  Range  -  3 

□  Transmission  Range  -  4  □  Transmission  Range  -  5  ■  Transmission  Range  -  6 

□  Transmission  Range  -  7  □  Transmission  Range  -  Nl  □  Transmission  Range  -  R 


3,574 

3,154 

5,358 


39,494 


17,762 
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Vehicle  B  (4mpg) :  Basic  Review  of  the  Sick  Vehicle 

•  Vehicle  fielded  to  this  guard  unit  spends  approximately  : 


»  29%  of  its  time  in  neutral. 
»  33%  of  its  time  in  7th  gear. 
»  40-45%  of  its  time  at  idle. 


»  3  to  5%  different  use  than  Healthy  Vehicle  A 


•  This  vehicle  has  numerous  active  fault  codes  some  of  which  do  not 
illuminate  the  malfunction  indicator  lamps. 

•  This  vehicle  has  network  faults  that  limit  the  data  collection 

•  This  vehicle’s  network  is  reporting  only  about  30%  of  the  time. 

•  This  vehicle’s  fuel  consumption  graph  is  skewed  and  indicates  that  the 
vehicle  is  consuming  close  to  50%  more  fuel  than  it’s  counterparts. 

•  When  first  instrumented  in  early  2008  these  vehicles  had  numerous  fault 
codes  of  which  the  operators  where  not  aware. 

•  Nearly  every  major  system  was  reporting  fault  codes 

-  Engine,  Central  tire  inflation  system  and  brakes. 
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Vehicle  B  (4mpg) :  Basic  Review  of  the  Under  Performing  Vehicle 

•  Both  vehicles  were  experiencing  injection  actuation  pressure 
system  faults  when  first  instrumented. 

•  Most  faults  occurred  at  higher  RPMs  (2k  +) 

•  Most  faults  occurred  before  the  engine  was  completely  warmed  up 
emphasizing  the  need  for  data  capture  on  startup 

•  Operators  reported  that  they  noticed  some  vehicles  seemed  to 
require  more  fuel  when  refueling  even  though  the  general  operation 
was  basically  the  same. 

•  After  repairs  at  depot  the  sick  vehicle  fuel  consumption  graph  now 
indicates  expected  operation. 
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Fault: 


Vehicle  B  :  Example  of  Fuel  System  Related  Fault 

IF1 


Engine  1  -  Injection  Actuation  Pressure  System  -  06/26/0 3  1 5:56:40  [LUG) 


Parameter: 


Engine  Fuel  Rate 


3 


The  red  line  indicates  a  fault.  Click  on  the  fault  to  display  additional  information. 


Parameter 

Wheel -Based  Vehicle  Speed 
Engine  Co olant  Tempe ratu re 
Road  Speed 

Engine  Coolant  Temperature 
Engine  Speed 

Accelerator  Pedal  Position  1 


Value 

Units 

12.785156 

kph 

47.0 

C* 

15.0 

kph 

116.0 

F° 

2140.375 

rpm 

86.8 

% 
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Vehicle  B  :  Corrected  Fuel  Consumption  After  Repairs 


Parameter: 


Engine  Fuel  Rate 


View:  Time  (s  e  cd  nds )  [T] 


Engine  Fusl  Rat*  (L/tir) 


100 
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Our  Family  of  DIME  Products 

Diagnostic  Information  Management  Environment  (DIME) 
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What  is  the  DIME? 


The  DIME  (Diagnostic  Information  Management  Environment)  is 
a  TRL  8  end-to-end  Lifecycle  and  Vehicle  Health  Management 
System  complete  with  on-board  storage,  GPS  tracking,  wired  and 
wireless  communication  interfaces.  DIME  enables  Connected 
Vehicle  applications. 

The  system  is  capable  of  remotely  updating  and  upgrading  the 
system  firmware  and  application  code  within  seconds  even  in  low 
communication  bandwidth  environments.  This  is  accomplished  via 
the  DIME  Data  Management  Center  where  data  is  stored,  analyzed, 
distributed  and  displayed. 
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What  is  the  DIME? 

HARDWARE  Features: 

•  Technical  Readiness  Level  (TRL)  8 

•  Ultra  low  power  consumption  (<25mW)  in  sleep  mode  with 
optional  ZERO  power  draw  configuration. 

•  Low  space  and  weight  claim  (approx  1 .5  lbs) 

•  Capable  of  start  up  and  recording  in  less  than  50ms 

•  802.1 1  b/g  wireless  interface 

•  2  CAN  channels  capable  of  1  Mbps  communications 

•  Ultra  low  cost  hardware  investment 

•  RS485  /  RS422 


•  Capable  of  waking  up  from  up  to  eight  unique  input  sources 
(CAN  Bias,  CAN  Activity,  Ignition,  J1708,  RS232,  RS422/RS485, 
External  input,  Real  Time  Clock) 
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What  is  the  DIME? 

HARDWARE  Features  continued: 

•  RS232 

•  Ethernet 

•  GPS  (Global  Positioning  System) 

•  Sensor  interfaces  (Analog  and  Digital) 

•  Remote  management  /  disablement  of  GPS/Wireless  comms 

•  CAISI  compliant  via  Ethernet  connection 

•  Forms  the  foundation  for  any  diagnostic  system  for  any  vehicle 
platform. 

•  Compatible  interface  with  mounted  and  mobile  vehicle  display 
systems. 

•  Disposable  technology,  low  logistics  footprint 
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What  is  the  DIME? 

Embedded  Software  Features: 

•  Remote  configuration 

•  Data  compression  techniques  enabling  performance 
in  low  bandwidth  Military  environments. 

•  System  processing  of  data  from  multiple  data  bus  sources 
(CAN,  J1708,  RS485)  simultaneously. 

•  Capable  of  storing  data  in  emergency  power  loss  situations. 

•  BIST  (Built  In  Self  Test) 
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THANK  YOU 


DRIVE  Developments,  Inc. 

7314  19  Mile  Road 
Sterling  Heights,  Ml  48314 
248-613-6738 
1 -877-705-DRIVE 

Larry  Osentoski 
President  and  CEO 
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Background 


*  2008  NDIA  Systems  Engineering  Conf 


—An  Air  Force  S&T  Directorate’s  View  on 
Applying  Systems  Engineering  (SE) 
Principles  to  its  Programs 


—  Introduced  an  ongoing  effort  to 
instantiate  the  practice  and  thinking  of 
SE  in  an  early  R&D  organization, 

*  Process  that  is  Streamlined,  tailorable  and 
flexible  to  apply  the  depth  needed  to  the 
specific  problem 


Materials  &  Manufacturing  Directorate 


This  Year’s  Focus:  Culture  and 
Community 

*  Share  thoughts  on  Culture 

—  The  last  thing  to  truly  change  in  a  transformation 
is  the  Culture 

—  Our  Team  has  a  foundation  in  the  streamlined 
process 

—  Task  at  Hand  is  to  get  Systems  Engineering 
into  routine  use  by  lab  Scientists  and  Engineers 
(S&E’s) 

•  Challenge:  Many  Laboratory  people  view 
Systems  Engineering  as  Acquisition  oriented 
and  stifling  to  creativity 


Materials  &  Manufacturing  Directorate 


What  is  the 
Culture 
of  a 

Laboratory? 


Materials 


Manufacturing  Directorate 


Lab  Demographics 


PERSONNEL 


DEGREES  &  SPECIALTY  AREAS 

76%  Civilian/Militarv  Scientist  &  Engineers  (S&E) 


Government 

87  %  Civilian 


13%  Military 


\L 


Contractors  (S&E,  Technician,  Ops  Support) 

Contractor  to  Govt  1.2  :  1 

19%  Contractors  have  PhDs 


38% 


31%/ 


Materials  Engineers 
Chemists/Chemical  Engineers 
Research  Physicists 
s>~  Aero  Engineers 

Safety/Environmental  Engineers 
Civil/Industrial  Engineers 
Biologists/Microbiologists 
Mechanical  Engineers 
Electrical  Engineers 


Typically,  70%  are  Task  Oriented  Personalities 

-  70%  of  those  task  oriented  personalities  are  Drivers 
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Changing  Culture  of  the  Lab 


*  Changing  from  “Performance”  objectives  to 
“Capabilities”  Focused  objectives 

*  Continue  to  Restructure  the  Organization 

*  Emphasizing  Integrated  Programs  with  other 
organizations 

*  Increased  competition  for  resources 


Moving  toward  a  prioritization  of  the  entire  portfolio 


Materials  &  Manufacturing  Directorate 


Range  of  R&D  at  the  Lab 


*  Basic  Research  to  Advanced 
Technology  Development  (ATD)  and 

Manufacturing  Technology 

(6.1  -  6.3  type  of  Funding) 

•  AFRL  Designated  Core  Processes  (CP) 

—  CP-1  Generate  Understanding  of  S&T 
Opportunities 

—  CP-2  Deliver  Needed  Technology  Options 
—  CP-3  Innovate  Solutions  to  Urgent  Needs 


The  Lab 

Scientists  and 

Engineers 

deal  with 

everything 

from  Basic 

Research  to 

fielded 

warfighter 

technology 

solutions 


*  Focused  Long  Term  Challenges 


Educated,  Adaptable,  Very  Busy 
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What  then  is  our  Culture? 

“Laboratories  are  Different” 


•  Great  People 

—  Heritage  of  government  service,  asking  “What  does  the  Air 
Force  need?” 


—  Strong  history  of  emphasis  on  scientific  advances  and 
creativity 

—  In-Depth  relationships  have  been  built  across  organizations 
based  on  technical  expertise 

—  Tend  to  be  independent  and  self-guided 

9  Dealing  with  Dramatic  Changes 

—  Performance  Based  to  Capability  Based 

—  Many  Organizational  and  Technical  Variables 

—  Higher  HQ  policies  and  instructions  impinge  on  scientists’ 
view  of  mission 

9  Recently  faced  with  Constrained  Resources 
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How  do  we  respond  to  this 
culture? 


%  SE  Rigor 


“The  Conversation” 


◄ - M - ► 

Some  without  tools  Some  without  tools  or 

tools  tailored  to  need 
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Timeliness  of  Culture  Issue 


2009MEGADIRECT0RY 


a  Crossroads 


For  the  U.S.  military, 
the  future  looks  more 
confusing  than  ever 
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How  can  SE  Help  in  Such  a 
Culture? 


*  DoD  SE  emphasis  came  out  of  Acquisition  concerns 

*  Lab  folks  feel  applying  SE  to  S&T  seems  like  a  stretch 

*  We  Believe: 

—  Streamlined  process  fits  our  culture 

•  Focused,  Succinct,  Tailored,  Affordable,  Owned  by  the  SME 

*  Applies  across  the  Program  Life  Cycle,  but  EMPHASIS  on  the 
Program  Planning  phase  (Greatest  Benefit) 

—  Hands-on,  early  experience  “sells”  the  value  of  the  process  / 
methodology 

—  Learning  occurs  during  the  process,  the  process  is  an 
opportunity  for  discovery 

—  This  is  a  creative  activity 


Materials  &  Manufacturing  Directorate 


Our  Current  Approach 


Spiral  II 

5-Step  Streamlined 
Program  Planning 

“Plan  the  program  right” 


Planning 


Spiral  I 

8-Key  Question 
Program  Assessment 

‘Consistent  SE  Assessment  ...6.1,  6.2,  6.3,  ATD’ 


Executing 


nnmrnm 


1.  Form  Team 

2.  Determine  Requirements 

3.  Generate  Alternatives 

4.  Evaluate  Alternatives 

5.  Deliver  S&T  Plan 


S&T 

Program 

Plan 

(with  Tech 
Transition 
Document) 


SE  “Vee” 

1.  Customer 
2.  Requirements 
3.  Demonstration 
4.  Tech  Options 


8.  Transition  Plan 
7.  Program  Structure 
6.  Risks 

5.  Best  Approach 


Iterative 


m 


S&T  to 
Internal  Lab 
Customers 


oeVWetVtj 

^ransitj- 


°ning 


S&T  to 
External 
Customers 


Spiral  III  -  Sharing  -  Community  of  Practice 
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Current  Streamlined 
Systems  Engineering  Process 


Step  1 _  Step  2 _  Step  3 _  Step  4 


Form 

Determine 

Generate 

Evaluate 

Team 

Requirements 

Alternatives 

Alternatives 

Deliver 

S&TPIan 


Do: 

•  Define  Problem 

•  Identify  all 

stakeholders 

•  Establish  Team 

•  Define  requirements 

•  Define  tech 

challenges 

•  Define  S&T  Exit 

Criteria  (KPP  sets) 

•  Validate  with  customer 

•  Understand  applicable 

state-of-the-art  & 
near  term 
;  technologies 

•  Brainstorm 
different  solution 
approaches 

•  Com  pare  alternatives 

across  req’ts  /  S&T 
exit  criteria 

•  Solicitcustomer 

approval  for 
proposed  solution 

•  FinalizeAF  Problem 

/  Goal  /  Solution 
Objectives 

•  Prepare  for  intended 

action  course 

Document: 

□  Problem  Definition 

□  Team  Directory 

(include  roles  & 
responsibilities) 

□  Team  Charter 

(optional) 

□  Prioritized 
Requirement  Set 

-  Performance 
-Affordability 

-  Producibility 

-  Reliability 

-  Supportability 

□  S&T  Exit  Criteria 

□  Alternative 

Definitions 

□  Tech  Readiness 

Assessment 

□  Manufacturing 

Readiness 

Assessment 

□  RiskAnalysis 

□  ValueAnalysis 

□  Cost  Estimate 

□  Schedule /Key 

Milestones 

□  AF  Problem  /  Goal  / 
Solution  Objectives 
statement 

□  Program  Roadmap 

□  Action  Plan 

Based  on  S&T  IPPD  Process  (Version  3,  2002) 


Materials  &  Manufacturing  Directorate 


Approach  to  Affecting  the  Cultur 

Based  on  the  Streamlined  SE  process 

*  View  S&E  Program  Managers  as  “internal” 
customers 

—  Tailor  approach  for  each  specific  project 

*  Emphasize  initial,  manual,  self-directed  approach 

(Computer  can  be  a  distraction) 

—  Hands  on,  with  facilitated  guidance 
—  First  hand  experience 
—  Familiarity  and  ownership  of  process 


Materials  &  Manufacturing  Directorate 


Tools  to  Implement  the  Approach 


Ain  Force  Rese^rcIh  L^boRMORy 

MateriaIs  Awd  MANufAauRiNq  TEdiNoloqy  DiRECionAiE 

Guide  For  Applying  k 

STRE^wLlNEd  Sy&TEMS  ENCjlNEERlNC,  (SE)  AppRGACb 
To  PrOC|R(W  PlflNNlNCj 
SplRAL  2  of  tFie  AFRURX  SE  Initiative 
VebsIon  1 .0 

24Aw,uw  2004 


Air  Force  Research  Laboratory 
Miaterials  and  Manufacturing  Technology  Directorate 

Self-Sufficiency  Workbook  for  Applying  a 
Streamlined  Systems  Engineering  (SE)  Approach 
to  Program  Planning 

Spiral  2  of  the  AFRL/RXSE  Initiative 

Version  1.0 
24  August  2009 
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Tools  to  Implement  the  Approac 


*  Guidebook  -  Users  manual  for  the  Streamlined 
Process  (description  of  “What  Is  It?”) 


•  Workbook  -  Means  of  capturing  the  preliminary 
data  and  decisions,  (the  “How  to  Do  It”) 

—  Can  be  used  by  informal  team  or  individual 
Portfolio  /  Program  /  Project  Manager 

—  First  Evaluation  can  provide  basis  for  Approval  Decision 
to  proceed  with  Team  based  process  -  or  provide 
sufficient  information  to  the  PM  (Go/No  Go) 

—  Subsequent  Streamlined  Process  work  with  full  team 
results  in  detailed  project  definition  /  with  Action  Plans 
and  Proposals 


Materials  &  Manufacturing  Directorate 


Guidebook  Page  11 

H -  = 

Applying  AFRL/RX  Streamlined  SE  Core  Process 


Figure  3  illustrates  the  RX  Streamlined  SE  Core  Process  and  indicates  that  it  is 
a  relatively  simple  process  that  generates  five  products. 


Do: 

•  Define  Problem 

•  Identify  all 

stakeholders 

•  Establish  Team 

•  Define  requirements 

•  Define  tech 

challenges 

•  Define  S&T  Exit 

Criteria  (KPP  sets) 

•  Validate  with  customer 

•  Understand  applicable 

state-of-the-art  & 
near  term 
technologies 

•  Brainstorm 

different  solution 
approaches 

•  Compare  alternatives 
across  req’ts/S&T 

exit  criteria 

•  Solicit  customer 
approval  for 
proposed  solution 

•  Finalize  AF  Problem 
/  Goal  /  Solution 

Objectives 

•  Prepare  for  intended 
action  course 

Document: 

a  Problem  Definition 
a  Team  Directory 
(indude  roles  & 
responsibilities) 
a  Team  Charter 
(optional) 

□  Prioritized 

Requirement  Set 

-  Performance 

-  Affordability 

-  Producibility 

-  Reliability 

-  Supportability 

□  S&T  Exit  Criteria 

□  Alternative 

Definitions 

□  Tech  Readiness 

Assessment 

□  Manufacturing 

Readiness 

Assessment 

□  Risk  Analysis 

□  Value  Analysis 

□  Cost  Estimate 

□  Schedule  /  Key 

Milestones 

□  AF  Problem  /  Goal  / 
Solution  Objectives 
statement 

□  Program  Roadmap 
a  Action  Plan 

Figure  3.  AFRL/RX  Streamlined  SE  Core  Process 


Product  1  -  Problem  Definition  and  Team  Directory 
Product  2  -  Prioritized  Requirements  and  S&T  Exit  Criteria 
Product  3  -  Alternative  Solutions 
Product  4  -  Evaluation  of  Alternatives 
Product  5  -  S&T  Plan 
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Guidebook  Discussion 
Step  1  -  Form  Team 


Step  1  Description 

The  Program  Manager  schedules  a  Team  Orientation 
Meeting  to  review  team  member  roles,  ensuring  that  they 
are  understood  and  obtaining  commitments  from  Customer 


Step  1  Products 

Product  1  under  Step  1  of  the 
Streamlined  SE  Process  is  a 


Representatives  and  Key  Team  Members.  The  SE  Facilitator 


Problem  Definition  and  Team 


presents  the  SE  Core  Process  and  a  Project  Overview  by 
conducting  a  review  of  all  elements  of  Homewark  #0  with  the 


□rectory. 


team 

Homewark  #1  begins  with  the  SE  Facilitator  providing  a 
written  overview  of  the  Streamlined  SE  Core  Process  and 
the  project  or  program  as  documented  in  Homewark  #0  to  all 
Team  Members  for  their  review. 

Next,  all  Team  Members  prepare  warksheets  for  the  Air 
Force  Problem,  Requirements  and  S&T  Exit  Criteria  and 


HorrewDrk  1  is  the  initial 
Requirements  and  S&T  Exit  Criteria 
werksheets  (Form  1.1) 


then  provide  them  to  the  SE  Facilitator.  These  werksheets 


are  available  in  the  RX  SE  Self-Sufficiency  VWarkbook. 

The  SE  Facilitator  compiles  the  Requirements  defined  by 
team  members,  and  forwards  them  to  the  Program  Manager. 

Finally,  the  Program  Manager  obtains  initial  customer  inputs 
on  the  Requirements  developed  by  the  team. 
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Workbook,  Two  Approaches 

*  Individual  or  Informal  Initial  Review 

•  Full  IPT  Plan 


1 1 

Air  Force  Research  Laboratory 

Materials  and  Manufacturing  Technology  Directorate 

Self-Sufficiency  Workbook  for  Applying  a 
Streamlined  Systems  Engineering  (SE)  Approach 
to  Program  Planning 

Spiral  2  of  the  AFRL/RX  SE  Initiative 

Version  1.0 

24  August  2009 

Materials  &  Manufacturing  Directorate 


Workbook  Pg  4,  Initial  Revie 


Project  Exploration  Decision 


(PM)  Exploration  and  Info  Gathering 

•  White  Papers 

•  Presentations 

•  Initial  Discussions  with  SE  Facilitator 

•  Strawman  Description  of  AF  Problem 

•  Use  Form  0.1  ‘AF  Problem, 
Requirements,  S&T  Exit  Criteria’ 


Expanded  discussion  of  this  element  of  the  RX  SE  Core 
Process  is  available  in  the  RX  SE  Core  Process  Guide 
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Workbook  PM  Initial  Review 


Project  Exploration  Decision 

Form  0.1  AF  Problem,  Requirements, 

5&T  Exit  Criteria' 

Program  Manager: _ 

Worksheet  for  A F  Problem.  Requirements ,  S&7~  Exit  Criteria 

Provide  a  “Problem  Statement”  that  captures  major  issues  and  scopes  problem 
space.  What  is  the  Air  Force  problem  to  be  solved?  Just  1  or  2  Sentences. 


Materials  &  Manufacturing  Directorate 


Workbook  Pg  13 


Step  1:  Form  Team 


(PM)  Kickoff/Team  Orientation  Meeting 

•Assumes  Project  Approved  to  Use  SE  Streamlined  Core  Process 

•  Ensure  Team  Member  Roles  are  Understood 

•  (SE  Facilitator)  Presents  Core  Process  and  Project  Overview 

•  (PM)  Gains  Commitment  from  Customer  Rep  &  Key  Team  Members 


Expanded  discussion  of  this  element  of  the  RX  SE  Core 
Process  is  available  in  the  RX  SE  Core  Process  Guide 
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Workbook  pg  15 
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Summary  and  Conclusions 


Instantiation  of  SE  in  S&T  Culture  is  continuing 


—  2008  -  Streamlined  Process  and  Early  Applications 

—  Today  -  Hands  on  approach  to  reach  our  culture,  and 
enhance  the  disciplined  creativity  of  discovery 


*  Invitation  to  the  Community  (SE  and  NDIA) 

—  Very  little  literature  on  the  application  of  SE  to  this  S&T 
culture 

—  DoD  emphasizing  Communities  of  Interest 

—  We  have  a  “Systems  Engineering  in  S&T”  Technology 
Area  in  DoDTechipedia 

•  https://www.dodtechipedia.miI/dodwiki/x/UINkAQ 
—  Please  visit  and  continue  the  conversation 
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DoD  Techipedia  Screen 
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Systems  Engineering  Tailored  to  be  Effective  in  the  Culture  of  a  Laboratory 

The  Department  of  Defense  (DOD).  and  the  Air  Force  (AF).  is  emphasizing  the  value  of  systems  engineering  (SE)  at  all  levels  of  defense  development  programs.  The  Air  Force  Research  Laboratory  (AFRL)  has  directed  that  SE  principles  must  be 
applied  to  programs  in  science  and  technology  (S&T)  as  well.  The  challenge  is  that,  in  the  laboratory  culture,  some  people  view  SE  as  acquisition-focused  and  counter  to  the  creativity  inherent  in  many  aspects  of  research.  The  solution  which  is 
envisioned  is  to  place  the  SE  principles  in  a  format  and  context  compatible  with  the  culture  of  laboratory  people  in  order  to  gain  their  acceptance.  That  culture  is  one  of  highly  educated  and  very  busy  people  who  need  to  understand  the  value  of  any 
new  "requirement"  (which  is  how  they  will  look  at  SE)  and  will  desire  some  ownership  before  they  will  embrace  it  The  approach  is  to  provide  a  streamlined  process  and  the  education/training  to  tailor  it  for  use  in  the  unique  situations  encountered  in 
each  laboratory. 
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Achieving  Systems  Engineering 
Culture  in  a  Science  and  Technology 
Laboratory  Environment 


Presentation  to 

NDIA  Systems 
Engineering 
Conf 

26-29  Oct  09 
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Software  Test  &  Evaluation 
Summit/Workshop 

Review 


The  Summit/Workshop  was  facilitated  by  the 
NDIA  Systems  Engineering  Division’s 
Software  Industry  Experts  Panel  and  the 
Developmental  Test  and  Evaluation  Committee 


October  26-29,  2009 


12th  Annual  NDIA  SE  Conference 
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Basis  for  SW  T&E  Summit/Workshop 


•  NDIA  SE  Division’s  SW  Committee  report 
completed  in  September  2006 

-  Top  Software  Engineering  Issues  in  the  Defense  Industry 

•  Key  Theme  of  the  Report 

Current  approaches  for  acquiring,  developing,  verifying 
and  sustaining  software  enabled  systems  are  inadequate 
to  deal  with  the  complexities  of  a  dynamic  and  changing 
acquisition  environment. 

•  Requested  to  identify  top  five  issues 

-  Actually  came  up  with  seven 
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Top  Seven  SW  Engineering  Issues 

1 .  The  impact  of  requirements  upon  software  is  not  consistently  quantified 
and  managed  in  development  or  sustainment. 

2.  Fundamental  system  engineering  decisions  are  made  without  full 
participation  of  software  engineering. 

3.  Software  life-cycle  planning  and  management  by  acquirers  and 
suppliers  is  ineffective. 

4.  The  quantity  and  quality  of  domain-knowledgeable  software 
engineering  expertise  is  insufficient  to  meet  the  demands  of 
government  and  the  defense  industry. 

5.  Traditional  software  verification  techniques  are  costly  and  ineffective  for 

dealing  with  the  scale  and  complexity  of  modern  systems. 

6.  There  is  a  failure  to  assure  correct,  predictable,  safe,  secure  execution 
of  complex  software  in  distributed  environments. 

7.  Inadequate  attention  is  given  to  total  lifecycle  issues  for  COTS/NDI 
impacts  on  lifecycle  cost  and  risk. 
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Issue  5  -  Description 
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Traditional  software  verification  techniques  are  costly  and 
ineffective  for  dealing  with  the  scale  and  complexity  of 
modern  systems  discussion  points: 

-  Over-reliance  on  testing  alone  rather  than  robust  SW  verification 
techniques. 

-  Manual  testing  techniques  are  labor-intensive,  scale  poorly,  and  are 
unproductive  relative  to  the  large  investment  of  resources. 

-  Compliance-based  tests  do  not  adequately  cover  risks  or  failure 
conditions. 

-  Tests  are  over-documented  with  disproportionate  effort  on  detailed 
procedures. 

-  Education,  training,  certifications  are  inadequate  to  develop  effective 
test  skills. 
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Issue  5  -  Recommendation 
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Study  current  software  verification  practices  in  industry,  and 
develop  guidance  and  training  to  improve  effectiveness  in 
assuring  product  quality  across  the  life  cycle. 

-  Sponsor  a  study  of  state-of-the-practice  verification  and  testing 
approaches. 

-  Review/update  testing  policies  and  guidance  to  emphasize  robust, 
productive  approaches  that  maximize  ROI. 

-  Review  adequacy  of  verification  plans/approaches  early  in  the  acq. 
life  cycle. 

-  Emphasize  skilled  investigation  throughout  the  life  cycle,  based  on 
coverage,  risk  mitigation,  high  volume  automation. 

-  Strengthen  curricula,  training,  certifications,  career  incentives  for 
testing  roles. 
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Summit/Workshop  Objective 

To  recommend  policy  and  guidance 
changes  to  the  Defense  enterprise  to 
emphasize  robust  and  productive  software 
Testing  and  Evaluation  (T&E)  approaches 
in  Defense  acquisition. 


NDIK 
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Location  &  Attendance 


smM.TflTHimi  nimmvm  a  t>:  khhat.y 


•  Hotel:  Hyatt  in  Reston  Town  Center,  VA 

•  Dates:  September  15-17,  2009 
•110  Registered  Attendee 

-  9  no-shows 

-  Approx.  80  stayed  to  the  end  of  last  day! 

•  Better  than  expected  participation! 
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Day  1  Agenda 
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8:00  Introduction  -  Why  this  Summit/Workshop 
8:10  Government  Presentations 
9:50  Break 

10:15  DoD  Industry  Panel 
1 1 :45  Lunch  &  Speaker 
12:45  SW  Test  Industry  Experts 
2:25  Break 

2:50  SW  Test  Industry  Experts 
4:30  Adjourn 
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Day  2  Agenda 
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8:00  Re-Cap  Day  1 

8:10  DoD  Services  Panel 
9:45  Introduction  of  Workshops 
10:00  Break 
10:30  Workshops 

12:00  Lunch  &  Speaker 
1 :00  Workshops 
2:30  Break 

3:00  Workshops 
4:30  Adjourn 
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Day  3  Agenda 
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8:00Re-Cap  Day  2 

8:10  Introduction  of  Workshop  Leaders 

8:15  Presentation  of  Issues  and 

Recommendation  by  Workshop  Leaders 

9:45  Break 

10:00  Way  Forward  Discussion  &  Final  Q&A’s 

-  Final  Summit/Workshop  Product  defined 

1 1 :00  Adjourn 
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Speakers  Morning  Day  1 


Framing  the  DoD  Software  T&E  Issues 

-  Dr.  Ernest  A.  Seglie,  Chief  Science  Advisor,  DOT&E 

-  Mr.  Chris  DiPetto,  Acting  Director,  DT&E 

-  Ms.  Kristen  Baldwin,  Director  for  System  Analysis, 

OD,  DR&E 
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Speakers  Morning  Day  1 


Panel:  Framing  the  Industry  Software 

T&E  Issues 

-  Mr.  Edgar  Doleman,  CSC 

-  Mr.  Bruce  Casias,  Raytheon 

-  Mr.  Tom  Wissink,  Lockheed  Martin 
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Speakers  Afternoon  Day  1 
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•  Lunch:  Mr.  Paco  Hope,  Cigital 

-  Software  Security  in  Defense  T&E 

•  Dr.  Cem  Kaner,  Florida  Institute  of  Technology 

-  Challenges  in  the  Evolution  of  Software  Testing  Practices  in 
Mission-Critical  Environments 

•  Dr.  Adam  Kolawa,  Parasoft 

-  Software  Development  Management 

•  Mr.  Rex  Black,  RBCS 

-  Risk-Based  Testing 

•  Mr.  Hung  Nguyen,  Logigear 

-  Software  Testing  &  Test  Automation 
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Speakers  Morning  Day  2 


Panel:  Framing  the  Services  Software 

T&E  Issues 

-  Dr.  James  Steilein,  US  Army  Test  and 

Evaluation  Command 

-  Dr.  Steve  Hutchison,  Defense  Information 

Systems  Agency  (DISA) 

-  Mr.  Mike  Nicol,  Aeronautical  Systems  Center, 

Wright-Patterson  AFB 

Lunch:  Mr.  Richard  Kuhn,  NIST 

-  Combinatorial  Testing 
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Remainder  of  Day  2 


Workshops  -  Three  Key  Challenge  Areas  (KCA): 

1 .  How  Much  T&E  is  Enough 

•  Risk  considerations,  Installed  System  T&E, 
Instrumentation,  Reliability,  Completion  Criteria, 
Coverage  and  C&A 

2.  Lifecycle  and  End-to-End  Software  Testing 

•  How  does  SW  T&E  get  involved  in  early  development 
(i.e.  left-hand  side  of  the  V-model  and  l&T  deliverables 

3.  Changing  Paradigms 

•  Open  Architecture,  COTS,  SOA,  SoS,  SaaS,  Legacy 
plus  New,  Security 
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Remainder  of  Day  2 


Workshops  -  Four  Focus  Areas  for  each  KCA: 

1 .  Review,  revise,  improve  RFP  Language 
(Including  T&E  activities/deliverables  in 
Competitive  Prototyping) 

2.  Training,  Competency  Model,  Human  Capital 

3.  Policy,  Guidance  &  Standards 

4.  Tools/Automation,  Methodologies  &  Processes 
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Results  of  Workshop  -  Raw  Data 


Issues 

1 .  Workshop  #1-108 

2.  Workshop  #2-  51 

3.  Workshop  #3-  20 

Total  - 1 79 

Recommendations 

1 .  Workshop  #1  -  44 

2.  Workshop  #2-  29 

3.  Workshop  #3-  13 

Total  -  86 


Participants 

1 .  Workshop  #1  -  30 

2.  Workshop  #2  -  31 

3.  Workshop  #3  -  25 

Total  -  86 
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Results  of  Workshop  -  Raw  Data 

Recommendations  by  Focus  Area 

-  17  for  FA  #1  Revise/Improve  RFPs  &  T&E  Deliverables 

-  23  for  FA  #2  Training,  Human  Capital,  Competency 
Models 

-  22  for  FA  #3  Policies,  Guidance  &  Standards 

-  17  for  FA  #4  Tools/Automation,  Methodologies  & 
Processes 

-  7  for  FA  #5  Costs,  Software,  Studies,  Organization 
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Way  Forward 
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This  is  a  Joint  effort  of  the  NDIA’s  SE  Division  DT&E 
Committee  and  the  Software  Industry  Experts  Panel 

1.  Workshop  #1  Team  to  complete  Recommendation 
Generation  by  October  9  (Done) 

2.  In  parallel  with  the  Item  1  generate  draft  outline  for  the  SW 
T&E  Summit/Workshop  White  Paper  (Done) 

3.  Review  and  correlate  Workshops  1 , 2  and  3  issues  and 
recommendations 

-  Update  White  Paper  outline  if  needed 

4.  Generate  Initial  White  Paper 

-  Completion  goal  -  December  4,  2009 
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Q  &  A 
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SW  T&E  Summit/Workshop  Presentations: 

www.ndia.org/Divisions/Divisions/SystemsEngineering/Pages/Test_and_Evaluation_Committee.aspx 


October  26-29,  2009 


12th  Annual  NDIA  SE  Conference 


20 


NDIK 

National  Defense  Industrial  Association 


Raytheon 

Customer  Success  Is  Our  Mission 


Successful  First  AESA  Deployment 
through  Application  of 
Systems  Engineering 

Terry  Duggan 
Scott  Nichols 

Christopher  Moore 
29  October  2009 


Outline 


Raytheon 


■  Background 

■  Approach 

■  Systems  Engineering  Activities 

■  Results  of  Analyses 

■  Readiness  Assessment 

■  What  Happened 

■  Lessons  Learned 
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Background 


Raytheon 


■  Raytheon  developed  a  new  AESA  radar  for  the  F/A-18E/F 
aircraft  under  contract  to  Boeing  for  the  US  Navy 

■  After  completion  of  OPEVAL  and  training,  US  Navy 
planned  to  deploy  two  full  squadrons  of  12  jets  each  with 
new  AESA  radars  for  a  six  month  deployment  in  support  of 
OIF/OEF. 

-  One  squadron  on  the  USS  Reagan  deployed  from  San  Diego,  CA 
and  one  on  the  USS  Roosevelt  deployed  from  Norfolk,  VA. 

■  US  Navy/Boeing/Raytheon  Team  dedicated 
to  deployment  success! 
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Used  a  Systems  Engineering  Approach  to 
Address  All  Aspects  of  First  Deployments 


Raytheon 


■  Created  a  joint  team  of  Navy,  Boeing  and  Raytheon  representing  the 
various  disciplines  required  for  a  successful  deployment 

-  Squadron  Commanders,  pilots,  maintainers,  engineers,  software  engineers, 
field  support  technicians,  repair  management,  etc 

■  Determined  Success  Criteria 

-  Stability  of  hardware 

-  Tactical  performance 

-  Inputs  from  Commanders 

-  Inputs  from  Navy  Maintainers 

-  Inputs  from  USN  PMO/DAPML 

■  Visited  each  of  the  2  squadrons  on  each  coast  to  conduct  pre-deployment 
readiness  review/coordination  sessions 

■  Assess  all  Logistics  Elements 

■  Developed  Action  Plan 

-  Developed  a  readiness  checklist 

-  Spares,  repairs,  retrofits,  lETMs,  etc 

■  Worked  Plan 

■  Supported  Deployment 

■  Prepared  Lesson  Learned 
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Using  Systems  Engineering  Techniques 
Was  Able  to  Assess  the  Hardware/Software 
Readiness  of  the  New  Radar 


Raytheon 


■  Hardware  Readiness 

-  Evaluate  maturity  of  hardware  to  be  deployed 

-  Determined  the  minimal  configuration  of  each  radar  LRU  (WRA). 

-  Identified  hardware  that  needed  to  incorporate  retrofits  for  radars  to  be 
deployed 

■  Evaluate  Performance  of  Tactical  Software 

-  Analyzed  OPEVAL  and  training  data  of  various  mission  profiles 

-  Analyzed  data  on  various  tactical  software  releases 

-  Analyzed  current  problems/anomalies  reported  from  fleet  and  pilots 

-  Developed  procedure  to  work-around  critical  anomalies 

■  Evaluate  performance  of  BIT  software 

-  Ability  to  accurately  Fault  Detect 

-  Ability  to  accurately  Fault  Isolate 

-  False  Alarm  rate 
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Systems  Engineering  Assessment  of 
All  Logistics  Elements 

■  Maintenance  Concept 

■  Supply  Support 

■  Repairs 

■  Depot  Status 

■  Tech  Reps 

■  IETMS 

■  PHST 

■  Maintenance  Training 

■  Reliability 

■  Support  Equipment 

■  Tools 


Raytheon 
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Results  of  Analyses 


Raytheon 


Priorities 


H/W  List  of  Items 

Comment 

High 

Med 

Low 

1 

Antenna  Filter 

Special  tools  and  training  will  be  needed. 

Difficult  to  do  on  airplane 

V 

2 

Reload  Spares  with  H4  OFP 

GPP,  ARI,  PPM,  SNBC,  and  MFA/BSC  need  to  be  loaded  with  H4,  8P 

V 

3 

Check  busbar  torques 

Should  check  for  torque  value 

V 

4 

Spare  IRR  Attach  bolts 

Top  and  bottom  bolts  (different  bolts)  have  been  know  to  strip  and  bind 

V 

-  check  torque  value 

-  spare  bolts 

5 

MFA  Attach  bolts 

-  spare  green  loctite 

V 

6 

FC  Cables 

extra  set  of  FC  cables  for  IQ  and  AL  between  CISP  and  REX 

< 

7 

Spare  RF  Green  Y  Cable 

RF  cable  between  Antenna  and  REX 

V 

8 

MFA  Busbar  Screws 

see  parts  list 

V 

9 

PCU  Repair  kit 

upper  bolt  repair  |fit 

V 

When  the  MSS  l  u<5  fail,  th  "e  v  ways  to  fix.  t 

>wap  IRR  2-  complete  teardown 

of  the  IRR 

10 

Spare  IRR's 

(risk  of  break  if  maint  m  ^  !  pr  pe  y) 

P\ 

V 

11 

Cover  Hinge  Loctite 

CISP,  REX  RPS,  . 

WlJ 

V 

12 

WRA  Shipping  containers 

_ 

J 

nr 

< 

13 

BIT  Tool 

Laptop  which  contains  the  BIT  Tool  to  heir 

roubles  hoot 

V 

14 

Check  CAL  information 

Run  the  BIT  CST  and  TR  Element  tests  on  the  airplanes  to  trend  performance 

V 

Special  tools  and  parts  will  be  needed--1/4  Turn  Cover  Fasteners,  Attach  bolts,  EMI 

15 

Subrack  Spare  Parts 

gasket,  QD's  (module  and  subrack) 

V 

Some  A/C  were  having  issues  at  power  up  (FCAL  not  connecting)  and  Arrays  failing 

16 

Review  A/C  "Grey"  failures 

and  healing  itself.  It  hasn't  officially  failed. 

V 

17 

Cable  assembly  for  support  pan 

in  case  it  breaks 

< 

Mitigation: 

•  Recommend  a  spare  radar 

•  Provide  consumable  parts  for  deployment 
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AESA  Maintenance  Concept 


Raytheon 


Organizational  Level 


FIRST  SCM  Rep 
MOB/Embarkation 


t 


Base/Ship 

Supply 

(Spares) 


Ashore 

Afloat 


USN  Activity 
Industry  Activity 
NRFI  Asset  Flow 
RFI  Asset  Flow 
Requisition  Flow 


Depot  Level 


Failed  WRA 


Direct  ship  (as  needed) 


Repair  Agents/ 
Suppliers 


OEM 


RFI  WRA 


Commercial 

Warehouse 


Boeing  STL 
Asset  Managers 


Status 
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Developed  Plan  to  Address  Items  in  the 
Readiness  Assessment 


Raytheon 


■  Hardware  upgrades 

-  Plan  for  retrofitting  Hardware 

■  Software  Upgrades 

-  Identified  required  version  of  tactical  software  (OFP) 

-  Identified  issues  with  software  performance 

-  Published  new  instructions  for  Pilots  to  mitigate  or  eliminate  problems 

■  Generated  a  minimal  Spares  List 

-  WRAs 

-  Consumables 

-  Special  tools 

■  Ensured  adequate  repair  contract  in  place 

-  Arranged  for  surge  capacity 

■  Identified  Additional  Maintainer  Training 

■  Established  a  24  hour  help  desk 

■  Established  contract  for  tech  reps  to  go  to  sea 

■  Provided  list  of  required  Support  Equipment 

■  Communicated  plan  and  status  to  all  stakeholders  weekly/daily 


Page  9 


24  Hour  Help  Desk/Repair  Communication 
Flow  Process  Implemented 


Raytheon 


Reagan  (Nichols)  Roosevelt  (Moore) 
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Results  of  VFA-22  Squadron 
Deployment 


Raytheon 


■  Deployed  on  cruise  with  12  F/A-18F  aircraft 

■  6  month  deployment  (May  to  November) 

-  1,713  sorties  flown 

-  3,773  hours  flown 

-  19  Radar  Parts  (WRAs)  ordered 

-  137  Maintenance  Discrepancies  written  against  the  radar 


Radar  Reliability  Exceeded  Predictions  and  Maintainers 

Complained  of  Nothing  to  DO 
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Results  of  VFA-213  Squadron 
Deployment 

■  Deployed  on  cruise  with  12  F/A-18F  aircraft 

■  Over  7.5  month  deployment  (September  to  April) 

-  2,120  Sorties  flown 

-  6,536  Hours  flown 

-  24  Radar  Parts  (WRAs)  ordered 

-  245  Maintenance  actions  written  against  the  radar 


Raytheon 
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Lessons  Learned  -  The  Good 


Raytheon 


■  Communications  throughout  the  planning  &  deployment  crucial 

■  Getting  all  the  stakeholders  involved  early  led  to  better  planning  and  execution 

■  The  work-around  procedures  eliminated  the  previously  experienced  pilot  problems 

■  Had  the  right  set  of  Support  Equipment  to  perform  majority 
of  required  maintenance  actions 

■  Had  sufficient  spares  on  board 

■  Broken  Non-Classified  Hardware  was  quickly  removed  and  sent  back  to  Raytheon 
for  repair 

■  Additional  tools  and  consumables  were  useful 

■  Great  support  from  24  hour  help  desk 

■  Prior  to  deployment,  the  verification  of  spares,  consumables  and  support  equipment 
paid  huge  dividends  while  deployed! 

•  Inventory  discrepancies 

•  Incorrect  NIIN’s 

•  Wrong  location 

•  Missing  quantities 


Did  it  but  could  have  been  easier 
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Lessons  Learned  -  The  Not  so  Good 


Raytheon 


■  Was  very  difficult  and  time  consuming  to  perform 
pre-deployment  verification  of  spares,  consumables  and 
support  equipment 

■  Lacked  ability  to  remove  Integrated  Radar  Rack  without 
improvising  a  stand  and  using  extra  bodies 

■  Process  to  get  broken  classified  hardware  off  the  ship  and 
back  to  depot  was  inconsistent  and  slow. 

■  Didn’t  identify  all  the  consumables  that  were  needed. 

-  Missing  one  cable 
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VFA-22  &  VFA-213  and  AESA 


Raytheon 


■  First  AESA  squadrons  to: 

-  Complete  the  workup  cycle 

-  Fly  Combat  Missions 

-  Drop  ordnance  in  Combat 

-  Fire  the  gun  in  Combat 

-  Complete  a  successful  CVN  deployment 

-  Numerous  AESA  articles  written 

•  Defense  daily 

•  Stars  and  Stripes 

•  Sea  Power  Magazine 


A  Successful  Deployment  -  Setting  the  Standard 
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Question  and  Answer 


Raytheon 
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System  of  Systems 


Cliche?  Buzz  word? 


Any  characteristics  in  an  SoS  different 
than  a  system? 


Is  the  engineering  effort  in  an  SOS 
different  than  traditional 
Systems  Engineering? 

Welcome  to  the  debate. 
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■  System-of-Systems  Challenges 

■  Definition 

■  Characteristics 

■  Challenges  and  Example  Cases 

■  Implementation  Strategies/  Solution  Considerations 

■  Engineering  the  SoS 

■  Architecture  and  Patterns 

■  Interface  Management 

■  Test  and  Evaluation 

■  Agile  Development 


Summary 


AF1T 


Systems  Engineering  Case 

Studies* 


GPS 


A-10 


F-111 


C-5  Galaxy 


JASSM 


Hubble  Space  Telescope 


B-2  Spirit 


Peacekeeper 


TBMCS  (Theater  Battle 
Management  Core  Systems) 


In  work  /  In  plan 

-International  Space  Station 
-Global  Hawk 

-  KC-135  trainer 
-T-6A,  E-10 

-  MH-53J/M  Helicopter 


Classified 

cases 


2007,  08,  09 


*  Unclassified  cases  available  for  download  http://www.afit.edu/cse 
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■ 


SoS  Definition 


A  SoS  is  defined  as  a  set  or  arrangement  of  systems 
that  results  from  independent  systems  integrated 
into  a  larger  system  that  delivers  unique 
capabilities. 

-  Defense  Acquisition  Guide 


■  Maier  (1998)  highlights  two  characteristics  that  distinguish  the  SoS  from 
very  large  complex  monolithic  systems: 

■  1 .  Operational  Independence 

■  2.  Managerial  Independence 

■  Maier  (1996)  and  others  originally  stated  others  characteristics 

■  3.  Evolutionary  Development. 

■  4.  Emergent  Behavior: 

■  5.  Geographic  Distribution: 
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Lots  of  DoD  SoS  Examples 


Space  Community 

■  ...“single,  fully  integrated,  multi-INT  architecture” 

■  ...“Community-wide  architecture”  ...“ground  architecture” 

■  ...“overhead  enterprise  architecture” 

C4ISR  Community 

■  Small  Clusters  of  Systems  (U2  -  Datalink  -  DCGS) 

■  Air  Force  Constellation  Net 

■  Air  Force  Research  Lab’s  Layered  Sensing  concept 

■  Airborne  Electronic  Attack  (AEA)  SoS  Architecture 


Name 


Naval  Surface  Warfare  Center 
Dahlqren  Division 

Single  Integrated  Air  Picture 


Space  and  Missile  Systems 
Center 


Space  Radar 


Theater  Joint  Tactical 
Networks 


Theater  Medical  Information 
Systems  -  Joint 


Acroi 


NSWC 


SIAP 


SMC 


SR 


TJTN 


TMIP 


Name 

Acronym 

Owner 

Army  Battle  Command 

System 

ABCS 

Army 

Air  Operations  Center 

AOC 

Air 

Force 

Ballistic  Missile  Defense 

System 

BMDS 

Joint 

USCG  Command  &  Control 
Convergence 

C2 

Convergence 

Coast 

Guard 

Common  Aviation  Command  & 
Control  System 

CAC2S 

Marine 

Corps 

Distributed  Common  Ground 
Station 

DCGS  AF 

Air 

Force 

DoD  Intelligence  Information 
System 

DoDIIS 

Intel 

Future  Combat  Systems 

FCS 

Army 

Ground  Combat  Systems 

GCS 

Army 

Military  Satellite 

Communications 

MILSATCOM 

Joint 

Naval  Integrated  Fire  Control  - 
Counter  Air 

NIFC-CA 

Navy 

From  DoD  SoS  Engineering  Guide  vl.O 


SoS  Challenges 


Interface  Management 

LEaders»jp 


erf0 


Competing  Operational 
Demands  (LDHD) 


o$ 


capabilities 


Schedules 


,r>ce 

INTEGRATION  FUNDING 


Stakeholders 

-nts 


^"esf  anc/ 
^v3/uaf/on  Co^ 


& 


•ll.erne',\  Control 
BeqU,rrteitient 

I^apa9e  STANDARDS 


IBaniimdl 


Le^cy  /ssues 


Survey  Item 

Percent 

Leadership 

75% 

Requirements  Communication 

75 

Standards 

71 

Funding 

68 

Knowledge  Skills  and  Abilities 

54 

Aligning  System  Interdependencies 

50 

End  to  End  Mission  Threads 

46 

Configuration  Management 

39 

Changing  Environmental  Demands 

32 

Information  Access 

32 

Organizational  Alignment 

29 

Commitment 

25 

Understanding  Scope 

25 

Deconflicting  Schedules 

25 

Doctrine 

21 

Interdisciplinary  Teams 

21 

Conflict  Negotiation 

21 

Let's  focus  on  a  few... 
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MILSATCOM  (AEHF)  Interface 

Management  Case 


Space  Segment 


EHF  Protected 


Wideband 


Network 


Narrowband 


H 

05 

UFO 

UFO/E 

AEHF 

Interim  Polar 

Jm 

(hosted) 

Terrestrial 
Global  Info 
Grid  (GIG) 


Mission  Planning 


i  _ 

I  MILSTAR 


AEHF 

WIPE 


CNPS/ODOCS 


l 


Navy  I  Joint/Others 


/£HF:  Aduarced  EHF  FAfl-T  Fsnfj-  to  feyord  Lue-of  a^l  TaminaU 

ACWS:  GB5L  !M^:roa™tEy*gtTi 

PIT  ArtjnmgLss^Twnhii  GMF  GfTHjml  MbMa 

CCS-C  GSODE  OapF  liar  Eal?  lie  Cmd  H  Ctrl  Etemert 

0£CS:  Defense  SaHHfi  CumiLnioaJMi  Spkin  GMT  {'-irraj-td Mu!i  limil  Tramnal 

EPS:  Enhanced  PctarSaWiite 


HER  RF:  Hgn  Data  Fait  Radio  Fluency 

MPE:  Mils*  on  Plafrmg  Ek“Tienl 

SMCS  rio  felts:  Mr-SNU  I  Coilfd  KkbKpAin 

FSAT-  Fror  TOTTuiDn^  Comm  Jiiatisra  RitnJ  hs  Syg 

TMQ5  IGAT  Msmi  OpsrFtai*  Syifem 

vm  WfeMGa|«jtr5Jtitfe 
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AF1T 


Interface  landscape 


Legend 


O  ICD 

I  I  Spec 


Externally  Controlled 
AEHF  Controlled 


Satellite 

Constellation 


Env  Spec 


SV  Spec 
SR-3100 


Milstar 

Satellites 

SR-2100 


AEHF 

Satellites 


( 


KI-54 

~r~ 


KI-54 

ICD 


PL  Spec 
SR-3120 


Crosslink 

SI-3055 

S/C-EELV 

SIS 

~r 


y 

y 


( 


S/C-PL 

SI-3115 


S/C  Spec 
SR-3110 


z 


EELV 


AFSCN 


ccs-c 


Milstar 

PL-Term 

SI-1135 

SI-2035 


Milstar 

Terminals 


ORD II 


TRD  10.0 


System  Spec 
SR-3000 


SGLS 
(SIS00502) 
and  USB 

l 


PL-Term 

SI-3135 

ICD1-3 

MCS/ 
Space  ICD 
SI-3125 
Vol  1 

PL 

Planning 

Constraints 

SI-3140 


MCS/  Space 
ICD 
SI-3125 
Vol  2 


} 

)r| 

- 

r 


MCS-CCS-C 
SI-3242 

-3465  Externa 
Reports 


Term  Rqmts 
Appendix 


NAST-T 


Joint  Term 
AEHF  Spec 
SR-3300 


MCS  Spec 
SR-3200 


MPE  Spec 
SR-3240 


MOPS  Spec 
SR-3210 


OSSE  Spec 
SR-3230 


TTSE  Spec 
SR-3220 


MCS-Term 
SI-3145 
Vol  1,  2,  6 

Terminal 

Planning 

Constraints 

SI-3430 

MCS-Term 
SI-3145 
Vols  3, 4,  5 


AUST-T 


Navy 

Terminal 


Army 

Terminal 


Air  Force 
Terminal 


TDSPP 


-< 


NAEDSS 


SI-3470 

MPSS-MPSS 


Army 

Terminal 


f  SI-3455  Vol  1  V- 
V  MCS-KMSS  / 

KTE 

/  SI-3455  \ 

<_Vol2  — 

NSA&UFO 

SOM 

/  SI-3415  \ 

\  Voll&2  /  * 

Factory 
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Cost  of  Interface  Management 


In  a  3  year  period,  56%  of  baseline  modifications  were  ICD-related 
$31. 5M  of  $71. 2M  (44%)  of  contract  modifications  were  ICD-related 


8.00 

6.00 

183 

0.00 

8.00 


Program  Change  Cost  vs  ICD  Category 

16.82 


2.00 

0.00 


1  1  R"? 

1  1  .U  r 

7  Q4 

0  00  0  31  0  07  U.bl 

/////  /'  ✓ 

cT  &  ^  Related  III 


(S' 

'V-  rV 


«'  /V 


&  JF 
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Percentage  of  Contract  Modifications  by  Catagory 

-12% 

9%  I  _i% 

-4% 

-4% 

14%  M2% 


44% 


□  Crosslink  (SI- 30 55) 

□  MCS  to  CCS-C  (SI-3242) 

■  MGS  to  SV  (SI3125) 

■  general  interface  update 


■  KI-54  to  SV  (Sl-xxxx) 

□  MCS  to  Terminal  (SI  3135) 

□  PL to  Terminal  (SI  31 45/3430) 

□  Non- interlace  related 


Case  Observation 

■  Cost  and  Effort  of  SoS  Integration 
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U-2  SoS  T&E  case 


■  Operational  concern: 


U-2S  aircraft 

Upgraded  SYERS-2A 

-multispectral  (EO/IR)  sensor 

Dual  Data  Link  2  (LOS/  BLOS) 

Distributed  Common  Ground  Station 


■  Test  events  being  planned  without  full  coordination 

■  T&E  plans  not  fully  validated 

■  Missing  opportunities  to  “piggy-back”  test  objectives 


■  Examined  Force  Development  Evaluation  T&E  Process 
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U-2  SoS  T&E  case 


C2  System  Program  Management 
New  Acquisition  and  Modernization 

Aircraft  System  Program  Management  Test  Objective:  “Verify  new  SYERS-2A  sensor  end-to-end 
New  Acquisition  and  Modernization  operations  and  to  demonstrate  full  airborne/ground  segment 
Flight  Test  Facility  functionality  with  DLL2  in  available  configurations  and 

operational  representative  architectures” 
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AFIT 


SoS  T&E  case 


-  Case  Observations 

■  SoS  Integration  is  NOT  Built  Into  the  Process 
■“Seamless”  Seams  Among  Interdependent  Systems  still  Real 

■  Ability  to  Define  the  “Ends”  Disappearing 

■  Program  Priorities  Dominate 


DoD  T&E  Summit,  2004,  Dr.  Glenn  Lamartin 

■  Increasing  complexity  and  interdependencies  of  systems 

■  Exponential  growth  in  interfaces  (network  participants) 

■  Increased  requirements  for  T&E  (Evolutionary  Acquisition) 

Network  Centric  Warfare,  1996,  Alberts,  Garstka  and  Stein 

“Testing  systems  will  become  far  more  complex  since  the  focus  will  not  be 
on  the  performance  of  individual  systems  by  on  the  performance  of  the 
federation  of  systems” 
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SoS  Emerging  Solutions 


■  Importance  of  Architecture  across  the  SoS 

■  Focus  on  interfaces 

■  Architectural  Pattern 

■  Acknowledging  the  different  roles  for  SoS 

■  SoS  Integration  and  T&E  Lessons  Learned 

■  Systems  engineering  versus  SoS  Engineering/ Architecting 


Address  acquisition  management  issues 

■  Agile  development  methodologies 

■  Appropriate  contracting  strategies 
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Solution  -  Architecture 


Emphasize  Operational,  Systems  Engineering 

•  Top-down  Architecting  and  Architecture  frameworks 
(DoDAF,  Zachman,  TOGAF,  FEAF,  etc) 

•  Bottom-up  system  integration  for  new 
CONOPS  and  Capabilities 

•  Early  Architecture  Evaluation/  Analysis 

•  Define,  organize  and  communicate  interfaces 


DoD  Architecture  Framework 
Version  2.0 


DoDAF  V2.0 


Volume  1 :  Introduction,  Overview,  and  Concepts 
Manager's  Guide 
18  May  2009 


“The  greatest  leverage  in  system  architecting  is  at  the  interfaces 
...  the  greatest  dangers  are  also  at  the  interfaces!” 


—  Mark  W.  Maier  and  Eberhardt  Rechtin, 

The  Art  of  Systems  Architecting,  CRC  Press,  2002 
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Solution-Architectural  Patterns 


■  Architect  interfaces  at  all  levels  of  abstraction  for 
agility,  adaptability  (evolution)  and  growth 

■  Layers  and  “Bowtie”  architectural  pattern  for  SoS  agility* 

■  SAB  concept  of  “convergence  protocol”** 


Hides 

World 

r 

Applications 

complexity 

Wide  -< 

Well  defined 

Web 

Info  Infrastructure 

interfaces 

Independent 

evolution 

Internet 

Networks 

Flexibility 

DOD/IC 

Sensors 

Agility 

& 


Targeting 


Application 

Diversity 


Searches 


% 


°// 


(loose  coupler) 


ff*IP  (LC) 


(loose  coupler) 


ATM  Ethernet 


Data  link 
Diversity 


Dial- 

ca6/e  UP 


*  Rich  Bryne,  MITRE,  from  2008  NRO  Systems  Engineering  Conference 

**  Scientific  Advisory  Board  2004, 
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AELL  Solution  SoS  Integration /  T&E 


Annette  Krygiel’s  “Behind  the  Wizard’s  Curtain” 
SoS  Integration  (mid  1990s)  for 

■  Digital  Mapping  Agency 

-  Digital  production 

■  Army  Task  Force  XXI 

-  Digital  battlefield 
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Solution  -  SoS  Integration/T&E 


1.  Key  Activities  need  to  preceed  SoS  integration 

■  Architecture  and  architecture  compliance,  system  test 

2.  Robust  Testing  strategy.  Early,  incremental  and  iterative 
integration 

■  Build  a  little — test  a  little 

3.  Plan  for  substantial  difficulties,  significant  time  and  resources 

4.  One  site  facilitates  integration  and  test  of  SoS  components 

5.  Address  the  leadership  of  the  SoS  integration 

6.  Prototyping  the  SoS  provides  early  insight  to  ops  requirements 

■  Test  with  Operators 
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Solution  -  Engineering  for  SoS* 


1.  Translating  SoS  Capability  Objectives  into  High- 
Level  SoS  Requirements  over  Time 

2.  Understanding  the  Constituent  Systems  and  Their 
Relationships  over  Time 

3.  Assessing  Extent  to  Which  SoS  Performance  Meets 
Capability  Objectives  over  Time 

4.  Developing,  Evolving  and  Maintaining  an 
Architecture  for  the  SoS 

5.  Monitoring  and  Assessing  Potential  Risk  and 
Opportunities  on  SoS  Performance 

6.  Addressing  SoS  Reguirements  and  Solution  Options 

7.  Orchestrating  Upgrades  to  SoS 


*  From  DoD  SoS  Engineering  Guide  vl.O 


Engineering  an  SoS 
Two  SoS  extremes 


“DIRECTED”  SoS  “ACKNOWLEDGED”  SoS 

(TO  BE/  OBJECTIVE)  (TO  BE/  OBJECTIVE) 


\7 


Ops  Mission  Architecture 
+  Decompose  Segments/  Systems 


\7 

Lead  Systems  Integration  (LSI) 


New  Missions 
+  New  Capabilities 


Modify  +  New  Systems 
+  Integration/  Design/ Architecture 


Prime  (LSI  w/  subs)  |_SI  w /  multi  Primes 

Design  Control  (ACA)  Coord/Plan/Architect 


Baseline  Systems  (AS  IS) 
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Need  for  Agile/  Adaptability 


■  Changing  Requirements  across  the  SOS 

•  Add /  Subtract/  Move  (phasing) 

•  Clarify/  Definition  of  Requirements  based  on  Ops  feedback 

■  Changing  Schedule  across  the  SOS 

•  Move  work  requirements  (phasing) 

•  Deployment  to  sites/  Ops  tempo 

Star 

■  Changing  Interfaces 

•  Add  new  interfaces,  Changing/  Clarify  Definition 


One  PM  suggested  the  need  for  “Flexpoints” 


Solution  -Acq  Implications 


■  Organizational  (People) 

■  Experience  with  SoS  Strategies 

■  Experience  with  Agile  development  methodology 

■  Familiarity  (or  connection)  with  the  Domain  (system  type) 

■  Attitudes  -  collaborative,  communicative 

■  Development  Method 

■  Spiral  or  Iterative  Lifecycle 

■  Scrum  software  practices 

■  Ability  to  handle  CHANGE 
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Conclusion 


■  SoS  Lessons  can  be  learned  from  system,  enterprise 
and  SoS  case  studies 

■  DoD  policy  and  guidelines  now  reflect  the  changing  IT 
landscape  of  system  of  systems 

■  Leaders  have  predicted  this  changing  landscape  will  directly 
impact  engineering  activities 

■  Requirements  &  Acquisition  community  must  address 

■  Growing  program  interdependencies 

■  Greater  numbers  of  potential  changes  across  the  SoS 

■  The  ability  to  operational  test  (and  resource  those  tests) 

■  Organization  aspects  to  best  handle  SoS  challenges 
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Agenda 

■  Purpose  for  Devising  a  Systems  Engineering  Assessment 
Methodology 

■  Systems  Engineering  Assessment  Methodology  Overview 

■  Systems  Engineering  Case  Study 

■  Systems  Engineering  Assessment  Methodology  -  Potential 
Applications 

■  Summary 
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Purpose  for  Devising  a  Systems  Engineering 
(SE)  Assessment  Methodology 

■To  assess  the  effectiveness  of  systems  engineering 
activities  and  to  show  how  this  knowledge  can  assist  with 
planning  for  activities  on  current  and  future  programs. 


4PL 


Hnliwnl  Swurfly  Space 


3 


Distribution  restricted  in  accordance  with  title  page 


11/4/2009  2:18  PM 


SE  Assessment  Methodology  Overview  - 
Systems  Engineering  Method 


■  Logical  set  of  activities  to  be  accomplished  in  every 
System  Life  Cycle  phase 


Figure  adapted  from  “Systems  Engineering  Principles  and  Practice”,  Kossiakoff  and  Sweet,  2003 


Requirements  Analysis  -  Assemble  and  organize 
input  conditions  and  clarify,  correct,  and  quantify 
what  the  system  must  do 

Functional  Definition  -  Translate  requirements  into 
functions  and  define  interactions  among  functional 
elements 


Physical  Definition  -  Translate  functional  design 
into  hardware  and  software  components  and  select 
preferred  approach  to  best  balance  performance, 
risk,  cost,  and  schedule 

Design  Validation  -  Design  models  and  the  system 
test  environment  then  simulates  or  test/analyze 
system  with  the  models 
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SE  Assessment  Methodology  Overview  - 

System  Life  Cycle 


■  System  Life  Cycle:  divides  complex  system  development 
process  into  phases 


Figure  adapted  from  “Systems  Engineering  Principles  and  Practice”,  Kossiakoff  and  Sweet,  2003 

■  Needs  Analysis  -  Defines  the  need  for  a  new  system  and 
determines  if  there  is  a  practical  approach  to  satisfying  such  a  need 

■  Concept  Exploration  -  Examines  potential  system  concepts  and 
identifies  required  performance  and  feasibility  of  possible 
approaches 

■  Concept  Definition  -  Analyzes  a  number  of  alternative  concepts  in 
order  to  select  a  preferred  concept  that  will  be  developed 

- dPL 
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SE  Assessment  Methodology  Overview  - 

Phase  Sequence 

■  The  Systems  Engineering  Method  is  applied  iteratively  to 
each  phase  of  the  System  Life  Cycle 

■  This  example  shows  the  Concept  Development  phase: 


Systems 
Engineering 
Method  Step 

Concept  Development  Life  Cycle  Phase 

Needs 

Analysis 

Concept 

Exploration 

Concept 

Definition 

Requirements 

A:  Analyze 

E:  Analyze 

i:  Analyze 

Analysis 

needs 

operational  requirements 

performance  requirements 

Functional 

B:  Define 

F:  Define 

J:  Define 

Definition 

system  functions 

subsystem  functions 

component  functions 

Physical 

C:  Visualize 

G:  Visualize 

K:  Select 

Definition 

subsystem  technology 

components,  architectures 

components,  architecture 

Design 

D:  Validate 

H:  Validate 

L:  Simulate,  validate 

Validation 

needs,  feasibility 

performance  requirements 

system  effectiveness 

Figure  adapted  from  “Systems  Engineering  Principles  and  Practice,”  Kossiakoff  and  Sweet,  2003 
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SE  Assessment  Methodology  Overview  - 

Activity  Context 

■  Need  knowledge  from  the  prior  steps  for  current  step 

■  May  be  done  by  same  people/organization  or  others 

■  Accumulated  steps  provide  the  whole  picture 

■  Information  does  not  need  to  be  complete  to  start  next 
activity  steps 

■  Should  be  sufficient  level  to  support  initiating  the  next  activities 


Inputs 

from 

previous 

steps 


> 


Activity  Steps 


> 


Outputs  to 
support 
follow-on 
steps 
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SE  Assessment  Methodology  Overview  - 

Terminology 

■  Impact  refers  to  the  level  of  influence  on  the  program 

■  Impact  ^  Effort  :  Impact  does  not  necessarily  reflect  amount  of  effort 
by  contractor  and  program  office 

■  Assessed  impact  to  the  project/program 

■  Three  levels  of  impact  (High,  Medium,  and  Low)  as  determined  by 
Sponsor  and  Subject  Matter  Experts 

■  “Actual”  vs.  “Ideal”  Impact 

■  “Ideal”  impact  assumes  prior  steps  were  done  sufficiently  to  support 
informational  needs  for  this  step  and  effort  progresses  exactly  as 
originally  planned 

■  “Ideal”  varies  depending  on  intended  application  of  SE  Method 

■  “Actual”  impact  is  the  assessed  program  impact  of  the  systems 
engineering  effort 
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Systems  Engineering  Case  Study  - 

Assessment  Goals 

■  Understand  the  impact  of  APL  SE  actions  and  activities  on 
the  program  and  their  relationship  to  the  whole 

■  Devise  a  way  to  look  back  at  how  tasking  evolved  from  baseline 
plan  and  its  impact  on  the  effectiveness  of  the  program 

■  Conduct  assessment  of  activities  to  understand  why  unanticipated 
activities  occurred 

■  Provide  considerations  and  guidance  to  be  used  for  planning  and 
organizing  future  activities 
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Systems  Engineering  Case  Study  - 
Case  Study  Description 

■  Sponsor:  Air  Force  Space  Command  -  Space  and  Missile 
Systems  Center  (SMC) 

■  Initial  Tasking 

■  Systems  Engineering  -  Requirements  generation  and  integration  of 
pilot  program 

■  Intended  scope  -  Concept  Exploration  Phase  within  Concept 
Development 

■  Evolution:  as  tasks  progressed,  information  gaps  were  identified 
and  activities  shifted  (with  sponsor  concurrence)  to  address 
these  needs 

■  Evolution  within  a  program  is  anticipated,  to  a  certain  extent,  with  the 
discovery  and  realization  of  key  system  concepts 

■  In  this  case  study,  tasking  and  activity  changes  differed  from  what  was 
expected  with  standard  SE  program  evolution 

■  It  is  important  to  understand  why  unanticipated  activities  occur  to  help 
learn  and  improve  for  the  future 

- rtPL 
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Systems  Engineering  Case  Study  - 

Major  Activities 

■  Requirements  Generation 

■  Requirements  Definition 

■  Mission  Analysis 

■  Technology  and  User  Studies 

■  Modeling 

■  Prototyping 

■  Concept  Demonstrator 

■  Concept  Development  Testing  Environment 
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Systems  Engineering  Case  Study 
Requirements  Definition:  Ideal 


Systems 
Engineering 
Method  Step 

Concept  Development  Life  Cycle  Phase 

Needs 

Analysis 

Concept 

Exploration 

Concept 

Definition 

Requirements 

Analysis 


Functional 

Definition 


Physical 

Definition 


Design 

Validation 


A 


B 


Key: 


D 


High  Impact 


Medium  Irr^ct 


Low  Impact 


Anticipated  requirements  activity:  create  a  Technical 
Requirements  Document 


- API 
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Systems  Engineering  Case  Study  - 
Requirements  Definition:  Actual 


Systems 
Engineering 
Method  Step 

Concept  Development  Life  Cycle  Phase 

Needs 

Analysis 

Concept 

Exploration 

Concept 

Definition 

Requirements 

Analysis 

A 

E 

i 

Functional 

Definition 

B 

F 

j 

Physical 

Definition 

C 

G 

K 

Design 

Validation 

D 

H 

L 

Key: 

High  Impact 

Medium  Impact 

Low  Impact 

■  Tasked  to  develop  Technical  Requirements  Document  (TRD) 

■  Found  guidance  documents  lacked  needed  detail 

■  Added  Process  Flow  documents  to  supplement  Needs  Analysis 
information  and  provide  common  understanding  of  system  functions 
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Systems  Engineering  Case  Study  - 
Requirements  Definition:  Comparison 


Ideal 

Concept  Development  Life  Cycle  Phase 

Needs 

Analysis 

Concept 

Exploration 

Concept 

Definition 

Requirements 

Analysis 

A 

E 

1 

Functional 

Definition 

B 

F 

J 

Physical 

Definition 

C 

G 

K 

Design 

Validation 

D 

H 

L 

Key: 

High  Impact 

Medium 

Impact 

Low  Impact 

Actual 

Concept  Development  Life  Cycle  Phase 

Needs 

Analysis 

Concept 

Exploration 

Concept 

Definition 

Requirements 

Analysis 

A 

E 

1 

Functional 

Definition 

B 

F 

J 

Physical 

Definition 

C 

G 

K 

Design 

Validation 

D 

H 

L 

Key: 

High  Impact 

Medium 

Impact 

Low  Impact 

■  Early  Needs  Analysis  information  was  not  mature 

■  Shift  in  focus  needed  to  earlier  steps 

■  Resources  and  information  unavailable  to  properly  address  later 
steps 

■  Level  of  resulting  information  insufficient  to  support  follow- 
on  Concept  Definition  activities 

- - - API 
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Systems  Engineering  Case  Study  - 
Mission  Analysis:  Ideal 


Systems 
Engineering 
Method  Step 

Concept  Development  Life  Cycle  Phase 

Needs 

Analysis 

Concept 

Exploration 

Concept 

Definition 

Requirements 

Analysis 

A 

E 

i 

Functional 

Definition 

B 

F 

j 

Physical 

Definition 

C 

G 

K 

Design 

Validation 

D 

H 

L 

Key: 

High  Impact 

Medium  Impact 

Low  Impact 

■  Analyses  conducted  to  support  requirements  effort 
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Systems  Engineering  Case  Study  - 
Mission  Analysis:  Actual 


Systems 
Engineering 
Method  Step 

Concept  Development  Life  Cycle  Phase 

Needs 

Analysis 

Concept 

Exploration 

Concept 

Definition 

Requirements 

Analysis 

A 

E 

i 

Functional 

Definition 

B 

F 

j 

Physical 

Definition 

C 

G 

K 

Design 

Validation 

D 

H 

L 

Key: 

High  Impact 

Medium  Impact 

Low  Impact 

■  Provided  important  knowledge  to  support  requirements 
activities 

■  Helped  to  supplement  incomplete  Needs  Analysis 
information 

- API 
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Systems  Engineering  Case  Study  - 
Mission  Analysis:  Comparison 


Ideal 


Requirements 

Analysis 


Functional 

Definition 


Physical 

Definition 


Design 

Validation 


Concept  Development  Life  Cycle  Phase 


Needs 

Analysis 


A 


B 


Key: 


C 

D 

High  Impact 


Concept 

Exploration 


Medium 

Impact 


Concept 

Definition 


K 


Low  Impact 


Actual 

Concept  Development  Life  Cycle  Phase 

Needs 

Analysis 

Concept 

Exploration 

Concept 

Definition 

Requirements 

Analysis 

A 

E 

1 

Functional 

Definition 

B 

F 

J 

Physical 

Definition 

C 

G 

K 

Design 

Validation 

D 

H 

L 

Key: 

High  Impact 

Medium 

Impact 

Low  Impact 

■  Actual  impact  in  Concept  Exploration  was  relatively  close  the  ideal 
impact 

■  Mission  needs  were  unclear,  thus  analysis  had  to  address  earlier  steps 
in  Needs  Analysis  than  initially  intended 

■  Resulted  in  diminished  ability  to  address  Physical  Definition  and  Design 
Validation  steps 
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Systems  Engineering  Case  Study  - 

Summary  Tables 


Ideal  Summary 


LC  Phase 

1 

Needs  Analysis 

2 

Concept  Exploration 

/ 

Concept 

J 

Definition 

SE  Step 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

RA 

FD 

PD 

DV 

RA 

FD 

PD 

DV 

RA 

FD 

PD 

DV 

Cell 

A 

B 

C 

D 

E 

F 

G 

H 

1 

J 

K 

L 

Requirements 

mm 

■■ 

MSM 

L 

Mission  Analysis 

III 

1 

L 

Tech  /User  Studies 

haH 

L 

Modeling 

L 

— H 

Fm 

ph 

wm 

Concept  Demo 

L 

FH 

n 

L  |  I 

Concept  Testing 

L 

L 

MBi 

_ 

Actual  Summary 


LC  Phase 

1 

Needs  Analysis 

2 

Concept  Exploration 

/ 

Concept 

J  1 

Definition 

SE  Step 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

RA 

FD 

PD 

DV 

RA 

FD 

PD 

DV 

RA 

FD 

PD 

DV 

Cell 

A 

B 

C 

D 

E 

F 

G 

H 

1 

J 

K 

L 

Requirements 

L 

Mission  Analysis 

L 

L 

L 

Tech/User  Studies 

L 

L 

L 

Modeling 

L 

L 

Concept  Demo 

L 

1  1  |  L 

L 

Concept  Testing 

L  1 

1  L  i 
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Systems  Engineering  Case  Study  - 
Summary  Assessment 

■  Relative  to  Ideal  impact,  Actual  impact  overall  was 

■  Less  than  anticipated 

■  Especially  in  Physical  Definition  and  Design  Validation  steps 

■  Earlier  in  the  life  cycle  than  anticipated 

■  Provided  higher  impact  in  Needs  Analysis 

■  Identified  some  needed  information 

■  Uncovered  additional  questions  to  be  addressed  by  sponsor 
organizations 


■  Impact  to  Concept  Exploration  and  Concept  Definition 
phases  was  lessened  due  to  Needs  Analysis  phase 
deficiencies 

■  Information  was  insufficient  to  support  CE  and  CD  activities 

■  Efforts  diverted  to  the  Needs  Analysis  phase 
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Systems  Engineering  Assessment 
Methodology  -  Potential  Applications 

■  Program  Office  planning  and  tasking 

■  Help  to  identify  information  needs  and  potential  gaps 

■  Help  to  visualize  activities  and  what  makes  them  successful 

■  Map  each  activity  to  appropriate  step(s)  and  identify  information  that 
precedes  it  as  well  as  what  steps  it  supports  in  turn 

■  Coordination  of  efforts 

■  Can  be  a  common  means  of  coordination  between  organizations 

■  Set  expectations  for  inputs  and  outputs  for  task  activities 

■  Clarify  deliverables  impact  and  stakeholders 
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Summary 

■  This  Methodology  was  useful  to  visualize  the  effectiveness 
of  real-world  systems  engineering  activities. 

■  Expect  this  Methodology  to  be  useful  in  assessing  the 
effectiveness  of  other  programs  so  that  additional  lessons 
can  be  learned  towards  future  improvements. 

■  Anticipate  this  Methodology  may  provide  additional  insight 
to  sponsors  and  to  internal  SE  teams  in  assessing  what  is 
required  to  support  a  given  effort. 
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Outline 


*  New  SE  Policy  and  Legislation  Implications 

*  SE  Reorganization 

*  Defense  Standards  role 

*  SE  standards  activities 
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DoD  Instruction  5000.02 


Mandatory  Materiel 
Development  Decision 

Mandatory  Milestone  A 
for  all  “major  weapon 
systems”  requiring 
technology  development 

Mandatory  system-level 
PDR  and  CDR  with 
reports  to  and 
assessments  by  the 
Milestone  Decision 
Authority  (MDA) 

Strengthened  MDA 
certifications  at 
Milestones  A  and  B 


Department  of  Defense 

INSTRUCTION 


number  5000.02 
December  $,  2008 

~USD(AT&L) 


SUBJECT:  Operatic®  of  the  Defeme  Acquisition  System 
References:  See  Enclosure] 

1  SSSOSE.  This  Instruction: 

f°  ““P^DoDffirecnve  5000  01  IRef  „ 

~  iaw,,  X  Td  Budget  c«c21  nt ?*  tiie 

*  ld  N-d  in  Enclosure  1  &>■  »d  the 

b'  Establishes  a  simvkfwrl  _ 


^  1  or  aus  issuance 

1  iismsiiiiyAiBraK 

and  programs  shall 
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Summary  of  Legislation 


The  Weapon  Systems  Acquisition  Reform  Act  of  2009 
contains  provisions  that  will: 

*  Address  problems  with  unreasonable  performance  requirements  by  requiring 
DoD  to  reestablish  systems  engineering  organizations  and  developmental 
testing  capabilities;  make  trade-offs  between  cost,  schedule  and  performance 
early  in  the  program  cycle;  and  conduct  preliminary  design  reviews  before 
giving  approval  to  new  acquisition  programs; 

•  Address  problems  with  unreasonable  cost  and  schedule  estimates  by 
establishing  a  new,  independent  director  of  cost  assessment  to  ensure  that 
unbiased  data  is  available  for  senior  DoD  managers; 

*  Address  problems  with  the  use  of  immature  technologies  by  requiring  the 
Director  of  Defense  Research  and  Engineering  to  periodically  review  and 
assess  the  maturity  of  critical  technologies  and  by  directing  the  Department 
to  make  greater  use  of  prototypes,  including  competitive  prototypes,  to  prove 
that  new  technologies  work  before  trying  to  produce  them;  and 

•  Address  problems  with  costly  changes  in  the  middle  of  a  program  by 
tightening  the  so-called  “Nunn-McCurdy”  requirements  for  underperforming 
programs. 

Excerpts  from  Bill  Signing  Ceremony  Press  Release  -  May  22,  2009 
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Acquisition  Lifecycle  Comparisons 

Defense  Acquisition  Management  System,  May  12,  2003 


(Program  Initiation) 


Concept 

Technology 

System  Development  and 
Demonstration 

Production  and  Deployment 

Operations  and 

Refinement 

Development 

Design 

\ y  Readiness 

Review 

y.  Full-Rate 
\s  Production 
Decision 

Support 

Review 


Concept 

Decision 


Defense  Acquisition  Management  System,  December  8,  2008 


Defense  Acquisition  Management  System,  May  22,  2009 
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Operations  and 
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DoD  5000.02  and  PL  111-23  Change  the 
Early  Acquisition  Landscape 
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PDR,  PDR  System-level 

t0  \!?e  CDR  with  an 
MDA,  and  initial 
Post-PDR-  product 

Assessment  baseline  and 

a  Post-CDR 
Report  to 
the  MDA 


Decision  Review 


Renewed  emphasis  on 
manufacturing  across 
the  lifecycle 


Post-CDR 
Assessment 
by  the  MDA 
between 
EMD  sub- 


What  are  the  implications  of  these  changes  for  programs  and  SEs? 

How  can  systems  engineering  enable  the  program  during  this  early  phase? 

How  can  standards ,  handbooks/guides  assist  the  programs? 
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What  This  Means  for  Systems  Engineers 


*  Systems  engineering  is  now  recognized  in  law  as 
inherently  necessary  in  requirements  definition, 
development  planning,  and  early  acquisition 

•  Need  for  and  focus  of  all  engineering  in  the  “pre¬ 
acquisition”  phases  (Materiel  Decision  Analysis 
and  Technology  Development)  is  dramatically 
altered: 

-  Earlier  engineering  involvement  (well  before  Milestone  A) 

-  More  government  expertise  to  plan  for  and  oversee 
requirements  definition,  technology  maturation,  and 
competitive  prototyping  leading  to  fully  expressed  system 
design  (the  allocated  baseline)  at  the  system-level  Preliminary 
Design  Review 
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New  Challenges  for  Programs 


*  Need  for  Program  Office  formation  and  PM  skill-sets 
after  MDD  and  prior  to  MS  A 

*  Increased  importance  of  the  Technology  Development 
Strategy  (TDS)  (as  a  surrogate  Acquisition  Strategy)  at 
MSA 

*  Schedule  and  funding  shifts  -  EMD  into  TD 

*  Earlier  engagement  with  industry  and  different 
contracting  strategies  for  technology  maturation, 
competitive  prototyping,  data  rights,  PDR  before  MS  B, 
etc. 

*  Explicit  need  for  earlier,  formal  SE  process  application 
(e.g.,  data,  configuration,  and  risk  management) 
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DDR&E  Organization 
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Director,  Systems  Engineering 
(DDR&E  SE) 


Director, 

Systems  Engineering 
Steve  Welby 

Principal  Deputy 
Terry  J.  Jaggers 


System  Analysis 


—  Red  teaming 

—  System  trades 

—  System  complexity  analysis 

—  Modeling  &  Simulation  Coordination  Office 

—  Developmental  planning 

—  System  of  Systems  SE 

—  Program  Protection  Plan  Policy  & 

Guidance/Acquisition  Cyber  Security 

I —  SE  Research  Center 


Major  Program  Support 


—  Program  Support  Reviews 

—  System  Engineering  Plans 

—  Program  Auditing  functions 

—  OIPT/DAB/DSAB  Support 

—  DAES  Database  Analysis  and 

Support 

—  Measurements  and  Analysis 

—  T&E  Oversight  List 


Mission  Assurance 


—  SE  and  SW  Policy  and  Guidance 

—  Specialty  Engineering 

—  DoD  Standardization 
Program  Office* 

—  Workforce  Development 

—  Mission  Assurance  Assessment 

—  Acquisition  Reform  Planning  and 
Reporting  for  SE 


Responsible  to  provide  technical  support,  systems  engineering  (SE)  oversight,  program 
development  and  mission  assurance  certification  to  USD(AT&L)  in  support  of  planned 

and  ongoing  acquisition  programs 
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DOD  Standardization 
Executive  Realignment 


•  Transfer  from  OSD  Logistics  to  OSD  Systems  Engineering 

•  Why?  -  Weapon  Systems  Acquisition  Reform  Act  of  2009 
codifies  Director  of  Systems  Engineering 

-  Provide  systems  engineering  principles  &  best  practices  to  enhance 
reliability,  availability,  &  maintainability  of  defense  systems 

-  Specifications  &  standards  are  key  systems  engineering  process  inputs  to  define 
requirements 

-  Specifications  &  standards  are  key  systems  engineering  process  outputs  to 
establish  product  baselines  and  measure  compliance 

•  Benefits  of  transfer  -  Director,  Systems  Engineering  will  set 
DoD-wide  strategic  direction  for  standards 

-  Standards  are  a  key  foundation  of  systems  engineering 

-  Standards  reduce  risk  and  cost  in  programs 

-  Standards  document  &  communicate  lessons  learned,  interoperability,  and 
technologies  across  entire  sectors  to  form  a  common  understanding 
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Defense  Standardization  Program 


Standardization  Policy:* 

DoD  policy  is  to  promote  standardization  of  materiel, 
facilities,  and  engineering  practices  to  improve  military 
operational  readiness,  and  reduce  total  ownership  costs 
and  acquisition  cycle  time.  It  is  also  DoD  policy  to  state 
requirements  in  performance  terms,  wherever  practical,  and 
to  make  maximum  use  of  non-Government  standards  and 
commercial  technologies,  products,  and  practices.  To 
pursue  these  policies,  there  is  a  single,  integrated  Defense 
Standardization  Program  and  a  uniform  series  of 
specifications,  standards,  and  related  documents. 


*Find  out  more  by  selecting  the  Policy  link  on  the  DSP  web  site:  http://www.dsp.dla.mil/ 
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ASSIST-  Online:  Acquisition  Streamlining  and 
Standardization  Information  System 


*  Provides  access  to  current  information  associated  with 
military  and  federal  handbooks,  specifications  and 
standards  in  the  management  of  the  Defense 
Standardization  Program 

•  Includes  reporting  features  and  an  exhaustive  collection  of 
both  digital  and  warehoused  documents 

*  Is  the  official  source  of  DoD  specifications  and  standards 

•  Includes  international  and  US  commercial  standards  and 
guides  as  deemed  applicable  by  the  DoD  community 


Register  at  http://assist.daps.dla.mil/online/start/ 
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SE  Standards  Revitalization 


Background: 

SE  is  DoD  gatekeeper/manager  for  SE  [and  integration  with  SW  Engineering] 
standards,  specifications,  and  non-DoD  guides  in  the  ASSIST  data  base  and 

a  participant  in  international  and  national  standards  organizations  for 
development,  revision,  coordination,  and  adoption  of  these  documents 

[Note:  a  large  category  of  IT  items  in  another  functional  area  not  addressed  here] 
Objective: 

Update  and  maintain  the  SE  and  SWE  portions  of  ASSIST;  support  efficient 
adoption  of  new  and  revised  documents 

Plan: 

Define  SE  role  in  SE  and  SW  standards;  develop  and  publish  processes  for 
standards  activities  (development,  revision,  coordination,  adoption,  posting 

in  ASSIST) 
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SE’s  Role  in  SE  Standards* 


•  Gatekeeper  for  Functional  category  /area:  SE  Standards 
and  Specifications  (SESS) 

-  -360  active  documents  in  SESS 

-  20+  involve  SE  participation  as  ‘preparing  agency’ 

•  National  and  International  SE  related  Standards: 

-  Participate  in  standards  bodies,  as  appropriate,  to  develop/revise 

-  Coordinate  review  of  new  drafts/revisions  to  support  DoD  vote 

-  Coordinate  for  adoption  within  DoD  and  placement  in  ASSIST 

•  DoD  Standards  [related  to  SE]: 

-  Participate,  as  appropriate,  in  development/revision  of  DoD  documents 

-  Coordinate  DoD  drafts  for  acceptance  and  ASSIST  placement 

-  Coordinate  Component-nominated  documents  for  acceptance  as  a  DoD 
spec/standard  in  ASSIST 


*also  includes  Specifications,  Handbooks,  DIDs 
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SESS  Lead  Standards  Authority 
[LSA]  Responsibilities 


*  Approves  project  #s  from  services  requesting  to 
adopt,  update,  develop  new,  or  cancel  standards  / 
DIDs  /  handbooks  /  specs 

*  Coordinates  review  of  such  and  mitigates  issues 

*  Selects  other  documents  to  adopt,  update, 
develop,  etc. 

*  Ensures  appropriate  persons  are  involved 

*  Determines  if  certain  items  belong  in  SESS  or 
elsewhere 
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List  of  Key  SE  Standards  Bodies 


•  ISO/IEC  -  International  Standards  Organization/ 
International  Electrotechnical  Commission  [particularly  the 
Systems  &  SW  Engineering  committee] 

•  TechAmerica  -  new  [merger  of  GEIA,  ITAA,  +...] 

-  WGs  of  particular  interest  to  DOD/SE  are  SE,  CM/DM,  Logistics,  Safety, 
HIS,  Enterprise  Information  Management  &  Interoperability  [new] 

•  IEEE  -  Institute  of  Electronics  and  Electrical  Engineers 

•  ANSI  -  American  National  Standards  Institute  (a  US 
standards  accrediting  agency  and  a  source  to  purchase 
ISO/IEC  standards);  discounts 

•  NATO 

•  Others  [e.g.,  AIA,  AIAA,  INCOSE] 
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SE  Policy  &  Guidance 
Recent  Activities 


•  Adopted  ISO/I  EC  Standards  into  ASSIST 

-  15288  [SE],  12207  [SWE,  in  progress],  16085  [Risk],  15939  [Measurement], 

26702  [IEEE  1220:2005] 

•  Reviewed  ISO  drafts  [in  JTC1/SC7  for  Systems  &  Software  Engineering] 

-  24748-1 ,2,3  guide  for  Life  cycle  management  and  guides  to  1 5288,  1 2207 

-  29148  [Requirements  engineering], 

-  15026-parts  1-4  System  &  SW  assurance 

-  1 0303  STEP  -  Standard  for  exchange  of  product,  in  process 

•  GEIA/EIA  [TechAmerica]  activity 

-  EIA-649  STD  and  HDBK  [CM]  revision  in  draft 

-  GEIA-Std-927  [A  Common  Data  Schema  for  Complex  Systems]  in  draft 

-  GEIA-Std-0009  [Reliability  Program  Standard  for  System  Design  &  Manufacturing;  adopted] 

-  GEIA-Std-0007  [Logistics  Product  Data];  handbook  to  follow 

-  Mil-Std-973  [CM  -  although  cancelled  -  now  points  to  EIA-649] 

-  GEIA-859  [DM;  adopted] 

•  Under  development  in  SESS*  [DDRE/SE  approves] 

-  MIL-STD-189A  Reliability  Growth  Management 

-  Mil-Std-31000D  Technical  Data  Package 

-  Mil-Hdbk-7  Acquisition  Data  Management  -  in  revision 

-  SEMP  DiD  updated 
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OSD  Participation  in  U.S.  Technical 
Advisory  Group  (TAG) 


•  TAG  to  International  Organization  for 
Standardization/International  Electrotechnical  Commission 
(ISO/IEC)  Subcommittee  7,  Software  and  Systems  Engineering, 
Working  Group  7  (WG7),  Life  Cycle  Management 

-  Sharon  Vannucci,  OSD/SE  Primary  rep 

-  Eddie  Bauer,  Army,  DoD  alternate  representative 

-  Karen  Richter,  Institute  for  Defense  Analyses  primary  representative 

•  Participation  in  standards  meetings 

-  Two  U.S.  TAG  meetings:  Portland  in  April  2009  and  Pittsburgh  in 
August  2009 

-  As  part  of  U.S.  national  body  delegation  to  WG7  Interim  meeting  in 
Nanning,  China  in  Nov  2008 

-  ISO/IEC  15026  Editors  meeting  in  Osaka,  Japan  in  August  2009 

-  Next  WG7  Interim  Meeting  in  Peru  in  November  2009 

•  Nomination  by  the  U.  S.  National  body  as  co-editors 

-  Karen  Richter  of  all  4  parts  of  ISO/IEC  15026 

-  Eddie  Bauer  for  all  3  parts  of  ISO/IEC  24748 
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Policy  and  Guidance  Structure 


*  Note:  Currently 
Guides  are  not 
posted  in  ASSIST 


M&S  Guidance 

DoD  M&S  Mgt 
DoDD  5000.59 


DoD  M&S  VV&A 
DoDI  5000.61 

RAM  Guide 
RAM-C 

Rationale  Report 
Manual 

CM  Mil  Hbk  61A 


WBS  Mil  Hbk  881 


Other  Standards  & 
Mil  Handbooks 


DoD  5000.02 


Data  Flow: 


Defense  Acq  Guidebook 

Chapter  4 

SE 

Chapter  8 
Security 

Acquisition  Policy 


Policy-specific 
guidance  linked  to  . . . 


Safety/ESOH  Guides 


Contracting  for 
SE  Guide 

SOS  SE  Guide 


IMP/IMS  Guide 


MOSA  Guide 

Other  Design 
Consideration  Guides 


“Wall  Chart” 


SEP  Prep  Guide 


DM  Guide 


Functional 

Architecture 

Development 

Guide 


Tech  Review  Guide 


>  Planned  ; 
Extent 


PPP 

DoDI  5200.39 

Systems  Assurance 
Guide 


DTM  08-048 
Supply  Chain  Risk 
Management 


PPP  Prep  Guide 


CPI  Security 
Classification  Guide 


CPI  Identification  Tool 


PP  Contract 
Language 
Compendium 


...  all  other 
relevant 
guidance 

EW  and  C2W 
Countermeasures 
DoDD  3222.3 


DoD  IA 

DoDD  8500.01  E 


Interoperability  & 
Supportability  of  IT  & 
NSS  DoDD  4630.05 


Acq  Security-Related 
Policies  &  Issuances 
Tool 


http://www.acq.osd.mil/sse/pg/guidance.html 


Systems  Engineering  Policy,  Guidance  and 
Standards  -  -  Initial  Focus  Plan  of  Work 

Address  issues  and  update  for  WSARA  and  DoDI  5000.02*: 

*  And  also  recent  NDIA-SE  report  on  DOD/SE  Systemic  Root  Cause  Analyses  findings  and  associated  actions 


Director,  SE  Directive 


Coordinate  and  obtain  USD(AT&L)  approval 


Contracting  for  SE 


Expand  beyond  current  Milestone  B  focus 


Data  Management 


Provide  guidance  to  DoD  data  managers 


New  Guidance 


Fill  in  policy  implementation  voids 


SEP  Prep  Guide 


Reduce  internal  and  external  redundancy;  update 


DAG  Chapter  4 


Update  for  WSARA 


SE  Standards 


Revitalize  SE  involvement 
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Integration  of  Operational  Simulations 
with  Physics-Based  Models  for 
Engineering  Analysis 


Stephen  Guest 
Henson  Graves 


©  2009  Lockheed  Martin  Corporation 


October  2009 


Outline 


Engineering  Analysis  and  Requirements 

Integration  of  Operational  Simulations 
with  physics  based  models  -  an  effective 
approach 

Examples 

Use  of  MBSE  for  Integration 
Conclusion 


Engineering  Analysis  is  Performed  to: 


•  Understand  requirements 

•  Determine  feasibility 

•  Evaluate  proposal  design 

•  Solutions  verification 


May  be  for  new  systems  or  upgrades 


Results  of  Analysis 


Provide  quantitative 
prediction  of  behavior  with 
evidence  to  back  it  up 

Examples 

Aircraft  performance 
Sensor  analysis 
Mission  effectiveness 
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Problems  Encountered  in  Analysis 


System  behavior  is  complex  and  depends  on 
complex  interactions  with  the  environment 

Need  for  component 
models  with 
appropriate  levels  of 
fidelity 

Availability  of  test 
data  for  modeling 
and  verification  of 
certain  behaviors 


pjsU'ibtflsd  Ambient  Mun}In*aan 


X 


imaging  Sensor 
(Vfshto  -  tnfratod) 


Path  Attenuation 


So  lar/Luttar 
Illumination 


Path  Emissions 
and  Scattering  - 
fl-pafh) 


Sofar/Lunar 
Reflections  (l  direct) 


Amhtew 

Reflections 

(t-ambmnO 


Thermal  Emissions 
ft  thermal) 


Entity  Control  and 
Communication 
of  State  and  Surface 


SensorPrime  from  Presagis 


Operational  Simulations  Integratedj&rfch 
Physics  Based  Simulations 


•  Operational  simulations  integrated  with  physics 
based  simulations  include 

Behavioral  models  of  vehicles  interacting  with  3D 
synthetic  environment 

Physics  based  models  of  sensors,  atmosphere, 
ballistics,  etc. 

Results  -  perform  experiments,  collect  data,  do 
analysis 

Must  understand  fidelity  of  models 

And  aggregate  integrated  results  as  basis  for 
evidence 


^Problem  -  M&S  is  Not  Traditionally  at  the  Core 

of  Product  Development  Processes 

•  Simulations  of  air  vehicle  and  subsystem 
are  in  common  use,  but 

•  Without  integration  of  detailed 
avionics/system/subsystem  models  with 
operational  simulation, 

Simulation  results  are  not  available  in  time 
to  affect  the  design  process 


easibility,  Cost,  an4v\ccuracy  Have 
Changed  Drastically  Over  Last  Few  Years 

•  Hardware/Software  improvements 

Of  course  it  means  the  cliche  of  increased 
performance  and  lower  costs 

But  what  it  really  means  is  that  I  can  run  it  on  my 
laptop 

Increased  performance  provides  opportunity  for 
more  useful/higher  fidelity  models 

SDKs  and  XML  provide  wide  ranging  interface 
capabilities 


Sensor  Display^lrnuTation 


i 


First  exampje  -  sensor  slew 
performancearTalysis^^ 


We  needed  to  evaluate  a  sensor’s  capability 
to  track  a  target  for  a  given  mission/ scenario 

Couldn’t  acquire  detailed  sensor 
performance  data  for  the  analysis 

So  we  built  the  operational  scenario  and 
integrated  with  a  perfect  sensor  tracking 
model 


Sensor  simulattefhresylts^ 


Simulation  executed  and  sensor  locked  to 
target 

Platform  jitter  not  modeled  (important 
assumption) 

Motion  data  of  sensor  captured  and 
max/min  rates  and  accelerations  calculated 

End  results  indicated  that  the  basic  aircraft 
motion  does  not  drive  sensor  performance 
requirements  (although  stabilization  with 
platform  jitter  may) 


Sensor  track 


simulation 


Second  example -  sensor 
resolution  analysis 


We  wanted  to  get  a  feel  for  what  sensor 
resolution/FOV  was  needed  for  target 
recognition  based  on 

B Mission  profile 

Multiple  resolutions  and 
FOVs  (zoom/magnification) 

Sensor  degradation  (blur,  persistence,  etc.) 
Display  characteristics 


Resolution  Ana4ys+s-{€entyj^^ 


We  could  use  models  like  Johnson  criteria 
Certainly  valid  but  not  satisfying 
Provides  disconnected  subjective  results 

Or  just  build  a  sensor  simulation  and 
evaluate  the  results 


Provided  direct  results  -  but  can  still  be 
subjective  as  one  person  may  make  positive 
identification  while  another  may  not 


FOV  /  Slant  range  analysis 


Use  of  MBSE  for  Integration 


In  order  to  better  establish  the  link  between  system 
simulations  and  operational  simulations,  we  are 
investigating  the  use  of  MBSE  and  SysML  to  define  the 
interfaces  and  auto-generate  code 

This  helps 

Build  re-useable  and  detailed  executable  design  models 
cost  effectively 

Obtain  quantitative  analysis  results 
Incorporate  the  results  into  the  design  process 


Leveraging  SysIVHr+ntegfa^tori^^ 


•  We  used  the  simulation 
environment  configured  with 

Performance  model  of 
aircraft 

Sensor  models 

Mountains,  obstacles, 
weather, ... 

•  To  understand  and  refine  the 
requirements 

To  understand  design 
constraints 

f  Aircraft  performance 
Radar  performance 
In  specific  operational 


Aircraft  with  radar  altimeter  and 
terrain  detection  radar 


scenarios 


Integration  of  a  1553-Bus4A4ith-er:l:errain 

Following  Simulation 


and  integrated  with  Simulation 


...  behavior  defined  by  state  charts,  uses  message  table, 
with  statistical  assumptions  about  performance 


Conclusions 


The  integration  of  physics  based  simulations  within  an 
operational  simulation  can  be  practical 

Building  and  assembling  models,  executing  distributed 
simulations,  collecting  data,  and  performing  analysis 
within  an  integrated  environment  is  relatively 
inexpensive. 

Leveraging  an  MBSE  approach  simplifies  the  creation  of 
an  executable  architecture  and  reusable  frameworks 
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Acquisition  Workforce 
Challenges  /  Opportunities 


•  To  increase  the  success  rate  of  our 
acquisition  programs,  we  need  to: 

-  Better  equip  /  support  /  enable  the  workforce  to  perform 
successfully  and  meet  all  demands 

-  Mitigate  loss  of  skilled  /  experienced  workforce 

-  Successfully  compete  for,  hire  and  retain  talent 

-  Transfer  knowledge  /  expertise  to  new  generation 

-  Return  “inherently  governmental”  and  other  appropriate 
work  from  contractors  to  the  government  workforce 

-  Integrate  acquisition  workforce  planning  with  DoD  Total 
Force  Human  Capital  Planning 

-  Strategically  plan  and  resource  human  capital  initiatives  - 
develop  and  execute  a  workforce  development  roadmap 


NDIA  SE  Conference:  SE  Workforce  Development  Update 
10/29/09  Page-2 


UNCLASSIFIED 


2 


Technical  Management  Workforce 

Population 


Technical 

Management 

Workforce 
42%  of  the 
Total 

Acquisition 

Workforce 

population 


The  Defense  Acquisition  Community 

125,047*  Government  and  Military  Certified  Professionals 

Over  50,000  DoD  Professionals  in  SE,  T&E  and  PQM 

*  DAU  Data  Mart 

500,000+  Defense  Industry  Personnel 

a/o  30  Jun  09 
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Systems  Engineering 
Workforce  Layers 


Systems  Engineering  (SE)  Work  performed  by  Defense  Prime  /  Sub  Contractors  in 
development  and  execution  of  Acquisition  Programs 
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Notional  DoD  Systems 
Engineering  Workforce  Strategy 


25-35 


35-45  45-55 

Workforce  Age 


>55 
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SE  Acquisition  Workforce  Age  Demographic* 

and  Notional  Strategy 
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US  Army  RDECOM  Career  Program  16* 
Age  Distribution  Ages  21-71  only  -  2009 


*  CP  16  includes  all  Army  research 
and  development  engineers 
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US  Army  RDECOM  Career  Program  16 
Age  Distribution  Ages  21-71  only  -  Trend 


21  23  25  27  29  31  33  35  37  39  41  43  45  47  49  51  53  55  57  59  61  63  65  67  69  71 


2001 

2005 

2009 


Percent  of 
population  who 
are  military 
retirees 
2001-1.0% 
2005-1.5% 
2009-  2.2% 
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Projected  FY10-15  New  Acquisition 
Government  Workforce  Distribution 


•  20,000  new  government  billets: 

-  Contractor  Conversions  to  Government  =  1 1 ,000* 

-  New  Hires  =  9,000 

•  Projected  Career  Field  Distribution: 

-  Contracting  =  9,500 

-  PM,  SPRDE/SE&PSE,  PQM,  LCL,  BCEFM,  etc.  =  10,500 

•  Future  Workforce  Total:  126K  +  20K  = 

146K 

•  Future  Contractor  Support  =  40K  (vice  51 K 
currently) 


*  Projected  at  1 4  Apr  08  DAU  Workforce  Conference 
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Current  Systems  Engineering  (SE) 

Workforce  Picture 


•  Recent  Congressional  and  GAO  reports  cite  evidence  of  lack  of 
disciplined  systems  engineering  -  indicates  gaps  in  competencies 

•  Systemic  Root  Cause  Analysis  efforts  to  date  indicate  lack  of 
systems  engineering  skills  and  numbers  in  the  SE  workforce 

•  No  clear  picture  of  what  competencies  are  available  in  the  current 
SE  workforce 

•  SE  workforce  members  may  work  on  a  single  component  for  entire 
career  or  may  work  in  only  one  area  across  several  programs 

•  SE  experience  standards  for  certification  levels  are  only  specified 
as  number  of  years  spent  in  coded  acquisition  positions  in 
specific  career  fields  -  not  an  indication  of  real  experience 

•  Number  of  years  of  experience  for  current  certification  levels  is  too 
low  when  compared  to  industry 
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Current  DoD  SE 
Certification  Picture 


*  Separate  functional  career  fields/paths  with  little 
integration  of  competencies  -  SE,  PSE,  T&E,  PQM 

*  Stove-piped  approach  to  certification  =>  less  agile 
workforce 

*  SE  &  PSE  paths  allow  for  other  career  field 
experience;  T&E  and  PQM  do  not 

*  Job  rotational  assignments  are  not  often  utilized  / 
emphasized 

*  Certification  is  often  seen  as  a  check-off  list;  no  real 
meat  behind  what  a  certification  means 

*  SE/PSE  and  T&E  require  degrees 
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SE  Workforce  Challenges  / 
Opportunities 


•  What  competencies  are  needed  now  and  in  the  future  and  what 
gaps  exist  or  will  exist? 

-  What  kinds  of  Systems  Engineers  do  we  need? 

-  What  is  the  difference  between  Systems  Engineers  and  other  domain 
engineers? 

•  What  workforce  capacity  do  we  need  now  and  in  the  future? 

-  What  is  the  right  SE  workforce  size? 

-  How  many  SEs  are  needed  on  any  particular  program? 

•  What  is  the  near-term  and  long-term  workforce  capability  risk? 

-  How  can  we  manage  and  mitigate  this  risk? 

•  What  key  information  will  help  us  make  sound  Systems 
Engineering  human  capital  strategy  /  initiative  decisions? 

•  How  do  we  leverage  NDAA  852  funding? 

-  What  should  we  do  in  terms  of  Recruiting? 

-  What  should  we  do  in  terms  of  Training  /  Development? 

-  What  should  we  do  in  terms  of  Retention? 
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Workforce  Development 


•  These  initiatives  constitute  our  workforce 
development  roadmap 

•  Initiatives  can  be  grouped  under  government, 
industry  and  academia  categories 

•  All  initiatives  are  interdependent  -  each  initiative 
complements,  leverages  and  affects  many  other 
initiatives 

•  Each  initiative  supports  one  or  more  efforts  under 
recruit,  train/develop,  and  retain 
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Workforce  Development  Initiatives: 

Government 


•  Competency  Assessments  for  Systems  Planning,  Research, 
Development  and  Engineering  (SPRDE-SE)  and  Production, 
Quality,  and  Manufacturing  (PQM)  Career  Fields 

•  New  Four  Level  Certification  Standards  for  SPRDE-SE 

•  Science,  Technology,  Engineering,  and  Mathematics  (STEM) 
Strategic  Plan  Working  Group 

•  SE  Executive  Technical  Leadership  Course  (SERC  Research 
Topic) 

•  Mentoring  Workshop/Tutorial  at  NDIA  SE  Conference  October  26, 
2009 

•  Systems  Engineering  (SE)  Defense  Acquisition  Workforce 
Development  Fund  (DAWDF)  Initiative 

•  Government-to-Government  Workshops  with  Singapore  on  SE 
Competency  Models 

•  Occupational  Career  Code  for  Systems  Engineering  (with  DAU) 
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Workforce  Development  Initiatives: 

Industry 


•  International  Council  on  SE  (INCOSE)  Certified  SE 
Professional-Acquisition  (CSEP-Acq)  and  Future  Extensions 

•  National  Defense  Industrial  Association  (NDIA)  SE  Division 
Education  and  Training  (E&T)  Committee  Comparison  of 
Acquisition  and  Developer  Competency  Models 

•  Others? 
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Workforce  Development  Initiatives: 

Academia 


•  Body  of  Knowledge  and  Curriculum  for  Advanced  SE  - 
BKCASE  (SERC  Research  Topic) 

•  Defense  Acquisition  Workforce  Certification  Equivalency 
with  Naval  Postgraduate  School,  Air  Force  Institute  of 
Technology  and  Air  Force  Academy  for  DAU  SYS  Courses 

•  Air  Force  Academy  Preparation  Course  for  INCOSE 
Associate  SE  Professional  (ASEP)  Certification 

•  SE  Education  Symposium  April  2010  (Co-sponsored  with  Air 
Force  Academy) 

•  SE  Education  and  Training  Summit  2010 

•  Working  with  DAU  on  SPRDE-SE  and  PQM  Curriculum 
Currency 

•  Collaboration  with  Civilian  Universities 
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Benefits 


•  All  of  these  initiatives  directly  contribute  to 
“raising  the  bar”  for  Systems  Engineering  across 
the  board  by: 

-  Enabling  us  to  assess  the  entire  DoD  Systems  Engineering 
workforce  across  critical  competencies 

-  Enabling  us  to  better  determine  shortfalls  in  both  competencies 
and  workforce  size  at  all  levels 

-  Enabling  us  to  better  manage  workforce  development 
requirements  and  certification  standards 

-  Enabling  us  to  make  better  decisions  about  human  capital 
strategy  and  initiatives  for  the  Systems  Engineering  workforce 

-  Enabling  us  to  provide  acquisition  programs  with  the  quantity 
and  quality  of  Systems  Engineers  they  need  for  success 
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Questions? 
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BACKUP 
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Human  Capital  Initiatives 

(Defense  Acquisition  Workforce  Development  Fund  ^ 


1  Based  on  NDAA  Section  852,  Defense  Acquisition  Workforce  Development  Act 
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Draft  Science,  Technology,  Engineering  and 
Mathematics  (STEM)  Education  and  Outreach 

Strategic  Plan 


U.S.  DEPARTMENT  OF  DEFENSE 
DEFENSE  RESEARCH  AND 
ENGINEERING 


Vision 

A  diverse  world-class  ^ 
STEM  talent  pool  with 
the  creativity  and  agility 
to  meet  national  defense  needs 


SCIENCE,  TECHNOLOGY, 
ENGINEERING  AND  MATHEMATICS 
(STEM)  EDUCATION  AND 
OUTREACH  STRATEGIC  PLAN 


Mission 


Inspire,  develop  and  attract  the  STEM  talent  essential  to  create 
innovative  solutions  for  the  Nation’s  current  and  future  challenges 


Inspire 

A  Nation  of  students,  parents, 
teachers  and  the  public  inspired  to 
engage  in  STEM  discovery  and 
innovation 

Develop 

A  future  world-class 

STEM  workforce 
talent  pool 

Attract 

A  dynamic  and  innovative  work 
environment  in  the  DoD  that 
attracts  and  retains  world-class 
STEM  talent 

Deliver 

A  coordinated,  collaborative 
and  cohesive  set  of  DoD 

STEM  programs 

Objectives 

•  Increase  the  awareness  and  importance 
of  STEM  to  foster  discovery  and 
innovation. 

•  Provide  opportunities  and  resources  for 
learning  and  personal  growth  that  stress 
academics,  knowledge,  skills  and 
attributes  required  for  STEM  discovery 
and  innovation. 

•  Strengthen,  expand  and  enable 
communities  of  stakeholders  to  provide 
a  continuum  of  formal  and  informal 
education  programs  and  opportunities. 

•  Directly  engage  populations 
underrepresented  in  STEM  fields. 

Objectives 

•  Identify  current  and  future  workforce 
needs. 

•  Increase  the  diversity  of  participants 
in  STEM  programs. 

•  Build  a  portfolio  of  DoD  STEM 
programs  to  develop  the  desired 
competencies  of  the  talent  pool. 

Objectives 

•  Identify  programs  and  best  practices 
that  attract  and  retain  world-class 
STEM  talent. 

•  Ensure  a  DoD  workplace 
environment  that  attracts  and  retains 
world-class  STEM  talent. 

•  Strengthen  and  promote  the 
awareness  of  STEM -relevant 
opportunities  within  DoD. 

Objectives 

•  Develop  a  systematic  approach  to 
identify  STEM  education  and 
outreach  programs  across  the  DoD 
components  and  agencies. 

•  STEM  Development  Office  will 
provide  and  maintain  apublicly- 
accessible  inventory  of  DoD  STEM 
programs. 

•  Develop  a  STEM  inventory 
communication  strategy. 

DoD  STEM  Development  Office 

DoD  DDR&E  STEM  DEVELOPMENT  OFFICE-  10/21/09  -  Dr.  hmra  Adoljie  -  702-588-1479 
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Science,  Technology,  Engineering  and 
Mathematics  (STEM)  Education  and  Outreach 

Strategic  Plan 


•  Vision:  A  diverse  world-class  STEM  talent  pool  with  the 
creativity  and  agility  to  meet  national  defense  needs. 

•  Mission:  Inspire,  develop  and  attract  the  STEM  talent  essential 
to  create  innovative  solutions  for  the  nation’s  current  and 
future  challenges. 

•  Goals: 

-  INSPIRE:  A  nation  of  students,  parents,  teachers  and  the  public 
inspired  to  engage  in  STEM  discovery  and  innovation 

-  DEVELOP:  A  future  world-class  STEM  workforce  talent  pool 

-  ATTRACT:  A  dynamic  and  innovative  work  environment  in  the 
DoD  that  attracts  and  retains  world-class  STEM  talent 

-  DELIVER:  A  coordinated,  collaborative  and  cohesive  set  of  DoD 
STEM  programs 
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Professional  Growth  to  Program 

Management 


Program 
Management 
Career  Field 


Master 

(minimum  8  years  of  SE  experience) 


SPRDE-Systems  Engineering  Career  Field 
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The  Human  in  the  System: 

Integrating  the  Human  into  the  system 
Integrating  HSI  Tools  into  Systems 
Engineering 


Jennifer  McGovern  Narkevicius, 

PhD 

Jenius  LLC 
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Systems  Thinking 


•  Look  at  the  needed  outcome  and  solutions 
as  a  whole 

•  Look  at  a  system  as  a  dynamic  and 
complex  whole 

•  Have  all  the  contributors  participate  in  the 
design  and  implementation  of  the  solution 

•  Bring  all  the  required  perspectives  together 
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PROGRAMS 


•  Unprecedented  number  of 
high  value,  high  visibility 
programs 

•  Increased  attention  to  the 
part  of  humans  in  all  programs 

•  Cross-program  integration 
becoming  significant  issue 
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What  is  HSI? 


•  HSI  is  a  multi-disciplinary  strategy  for  the 
design  and  life-cycle  support  of  systems 

-  Based  on  human-centric  issues 

-  Executed  as  a  systems  engineering  activity 

-  Requires  unique  mind-set  to  system  design 

•  HSI  is  a  concurrent  engineering  process 


•  Main  concerns: 

-  Maximize  Total  System  Performance 

-  Minimize  Life  Cycle  Cost 
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TOTAL  SYSTEM  PERFORMANCE 


Operational  Availability 


JENIUS 

'  SoCutions 


Measurable  and  Certifiable 


an 
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HSI  Domain  Considerations 


Human  Factors 
Engineering 


Manpower, 
Personnel,  & 
Training 


Habitability  & 
Personnel 
Survivability 


Safety,  Enviro  & 
Occ  Health 
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There  is  no  such  thing 
as  an  Unmanned  System 


HSI  in  SE 


Define  user 
Define  user  intent 


Concept 
of  Operations 


High-Level 

Requirements 


Define  requirements  from  people 
Define  requirement  for  people 


Design  for  operator,  maintainer 
Incorporate  trade  off  decision 
information  based  on  human 
capabilities  and  limitations 


JENIUS} 
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Baited 

ements 


Ofic  ratio  ns  & 
Maintenance 


System 

Acceptance 


Collect  lessons  learned 


High  Level 
Design 


Detailed 

Desip 


Subsystem 

Verification 


Integration  &. 
Test 


Continuous  test/validation 


Implementation 


Time 


Identify  User  specific  requirements 

Functions  allocated  to  User 


f  Define 
/  User 
Customer  Needs 


Requirements 


Functional  Analysis  and  Allocation 


Process  Output  \ 

_ t 


Design  Tradeoffs  to  meet  the  need 


u  secumy  Suits  isis 


Manage  Requirements 

-  Requirements  traceability 

Maintain  control  of 
systems  architecture 
definition 


Measurement 

-  Decision  Support/Risk 
bias  mitigation 

-  Roadmap/Progress  map 

-  Risk  reduction,  mitigation 
tracking 
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HSI  Tools 


Requirements  Definition 


-  Top  Down  Requirements  Analysis/Top  Down  Functional 
Analysis 

-  Operation  Decomposition 

Management/Planning 

-  Human  Systems  Integration  Plan 

Measurement 

-  Domain  Specific 

Design  Definition  and  Refinement 

-  Operational  Sequence  Diagram 

-  Modeling  tools 

-  Prototyping  and  simulation 

-  Usability  Engineering  Process 


* 
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Top-Down  Requirements  Analysis 


Front-end  of  HSI  Process 

Provides  analyzed 
requirements, 
allocation  concepts, 
workload  estimates, 
human  task  models, 
system  metrics,  & 

manning  models 


Functional  Flow  Block  Diagram 


Functional  decomposition 

•  Traceable  to  requirements 

•  Temporal  sequences 

•  Links  system  level  elements  to  design  elements 


Modeling,  Prototyping 
and  Simulation 


File  Edit  Search  Display  Execute  ActionView  Analyze  Window  Help 


JMPlEqM-l 


pUpivpK.  3?|  Hlpflt, 
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Workload  Estimation  Algorithms 


n 

^1 

Hun 

nan  Performance 
Simulation 

Mockups 

Consider  all  of  the  design  factors  to  assess  the  implications  and_  suitability  of 
the  design  trades  made... Use  structured,  low-fidelity  assessments  with  trained 
users  to  identify  significant  issues  that  should  be  addressed  early 


JENIUS 

SoCutions 


Linkage  between  Tools 


HSI  TOOLS 

•  Requirements  Definition 

•  Management/Planning 

•  Design  Definition  and 
Refinement 

•  Measurement 


SE  TOOLS 

•  Requirements 
Management 

•  Management  Planning 

•  Maintain  control 

•  Measurement 
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Challenges 


Linking  tools  together 

-  Actually  passing  data  between  tools 

-  Without  losing  functionality,  information,  clarity,  “validity” 

Focusing  on  the  really  important  stuff 

-  Its  not  the  tool 

-  The  tool  is  just  a  tool... it’s  what  you  do  with  the  output  that  matters 

Maintenance 

-  It  is  not  enough  just  to  build  a  tool 

-  Tools  should  both  be  maintained  and  be  commercially  viable 

Tools  must  be  integrated 

-  Across  technical  disciplines 

-  Across  questions  of  technical  interest 

-  Across  phases  of  development 
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M&S  Practices 


DoD  Modeling  & 

Governance 


(M&S) 


M&S  Management  Structure  Organized  by  Communities. 
Designed  to  Support  &  Integrate  M&S  Activities  across  the  Department. 
Led  by  a  Senior  Level  M&S  Steering  Committee 
(M&S  SC)  to  provide  governance. 


Acquisition 

AT&L 


Analysis 

CAPE 
&  JS 


Experimentation 

JFCOM 


Intelligence 

USD(I) 


Planning 

JS 

&  Policy 


Testing 

DOT&E 
&  AT&L 


Training 

P&R 


Corporate  &  Crosscutting  M&S  Tools 


Corporate  &  Crosscutting  M&S  Data 


Corporate  &  Crosscutting  M&S  Services 


)  (AP\EXC 


(JCDE  EC) 


Components 
OSD,  Joint  Staff,  COCOMs,  Services 


Goal:  Establish  corporate  M&S  management  to  address  DoD  goals: 
Leads/guides/shepherds  the  $Bs  in  DoD  M&S  investments;  adds  value  thru  metrics  & 

ROI-driven  priorities;  and  seeks  to  provide  transparency. 
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Acquisition  M&S  Working  Group 

Relationships 


Industry 


Chair:  Jim  Coolahan 


DoD  Acquisition 


DoD  M&S 


Chair:  Col  Eileen  Bjorkman,  USAF 
SAF/XCDM 


Mr.  Mike  Truelove  (Ctr) 
Acquisition  Member: 
OUSD(AT&L)/DDR&E/SE/MA 


AMSWG  Charter  (SE  Forum,  2006) 

•  Assist  PMs  and  acquisition  professionals  by  improving  the  utility  of  M&S  .  .  . 

•  Address  common  concerns,  improve  info  flow,  align  technical  initiatives,  pursue 
cross-cutting  issue  resolution  .  .  . 

•  Represent  the  acquisition  community  in  DoD  M&S  deliberations  .  .  . 
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Acquisition  M&S  Master  Plan  Structure 


•  Foreword 

«  Introduction 

•  Purpose 

•  Vision 

Department  of  Defense 

•  Scope 

Acquisition  Modeling  and 

•  Objectives  (5) 

Simulation  Master  Plan 

•  Actions  (40) 

•  Action 

■  Rationale  (why  it’s  needed) 

Issued  by  the 

■  Discussion  (implementation  guidance) 

DoD  Systems  Engineering  Forum 

•  Lead  &  supporting  organizations 

April  17,  2006 

i 

■  Products  (what  is  expected) 

•  Completion  goal  (year) 

*  Execution  Management 
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AMSMP:  Five  Objectives,  40  Actions 


Objective  1 


Provide 
necessary 
policy  and 
guidance 


1-1  M&S 

management 

1-2  Model-based 
systems 
engineering  & 
collaborative 
environments 

1-3  M&S  in  testing 

1-4  M&S  planning 
documentation 

1-5  RFP  &  contract 
language 

1-6  Security 
certification 


Key 

Broader  than  Acqn 


Objective  2 


Enhance  the 
technical 
framework 
for  M&S 


2-1  Product 
development 
metamodel 

2-2  Commercial 
SE  standards 

2-3  Distributed 
simulation 
standards 

2-4  DoDAF  utility 

a)  DoDAF  2.0 
Systems 
Engineering 
Overlay 

b)  Standards  for 
depiction  & 
interchange 

2-5  Metadata 
template  for 
reusable 
resources 


Objective  3 


Improve 
model  and 
simulation 
capabilities 


3-1  Acquisition 
inputs  to  DoD 
M&S  priorities 

3-2  Best  practices 
for  model/sim 
development 

3-3  Distributed  LVC 
environments 

a)  Standards 

b)  Sim/lab/range 
compliance 

c)  Event  services 

3-4  Central  funding 
of  high-priority, 
broadly-needed 
models  &  sims 

a)  Prioritize  need: 

b)  Pilot  projects 

c)  Expansion  as 
warranted 


Objective  4 


Improve 
model  and 
simulation 

use 


4-1  Help  defining 
M&S  strategy 

4-2  M&S  planning 
&  employment 
best  practices 

4-3  Foster  reuse 

a)  Business  model 

b)  Responsibilities 

c)  Resource 
discovery 

4-4  Info  availability 

a)  Scenarios 

b)  Systems 

c)  Threats 

d)  Environment 
4-5  VV&A 


Objective  5 


5-2  Harvesting  of 
commercial 
M&S  lessons 


5-3  Assemble  Body 
of  Knowledge 
for  Acqn  M&S 

5-4  M&S  education 
&  training 

a)  DAU,  DAG  & 
on-line  CLMs 

b)  Conferences, 
workshops  & 
assist  visits 


a)  Documentation 

b)  Risk-based 


5  5  MSIAC  utility 


c)  Examination 


4-7  M&S  utility  in 
Acqn  metrics 


Acquisition  M&S  Community 
Recognizes  Importance  of  VV&A 


•  VV&A  of  M&S  is  important  because  it: 

-  Provides  an  understanding  of  the  assumptions,  capabilities,  and 
limitations  of  the  models  and  simulations 

-  Provides  a  means  for  knowing  how  trustworthy  the  M&S  results  are 

-  Promotes  reuse  of  M&S  by  allowing  others  to  understand  how  the 
M&S  has  been  used 

*  Three  specific  actions  for  VV&A  are  in  the 
Acquisition  M&S  Master  Plan  (AMSMP): 

-  Action  4-5  (a):  Require  DoD-wide  standardized  documentation  of 
VV&A 

-  Action  4-5  (b):  Develop  a  risk-based  methodology  &  associated 
guidelines  for  VV&A  expenditures 

-  Action  4-5  (c):  Examine  the  relevant  VV&A  when  M&S  informs 
major  acquisition  decisions.  Unambiguously  state  the  purpose,  key 
assumptions,  and  significant  limitations  of  each  model  or  simulation 
when  results  are  presented 
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Acquisition  Community 
Commitment  to  VV&A 


•  The  Acquisition  M&S  Community  has  followed 
through  by  taking  multiple  actions  to  promote 
and  encourage  VV&A: 

-  Sponsored  funding  to: 

-  Finalize  the  MIL-STD-3022  DoD  Standard  Practice  Documentation  of 
VV&A  for  Models  and  Simulations 

-  Finalize  the  DoD  VV&A  Documentation  Tool 

-  Establish  a  risk  based  approach  for  VV&A 

-  Develop  a  CMMI-like  VV&A  maturity  model 

-  Provided  guidance  on  VV&A  in  the  Defense  Acquisition  Guidebook 

-  Under  the  Acquisition  M&S  Working  Group,  established  a  VV&A 
Subcommittee 

-  Recommended  improvements  to  the  new  DoDI  5000.61 

-  Volunteered  to  participate  in  NATO  VV&A  activities 
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Acquisition  Commitment  to  VV&A: 

Sponsored  Funding 


•  MIL-STD  3022: 

-  The  effort  was  initiated  by  the  Navy  and  the  Acquisition  Community  saw 
value  that  benefited  the  M&S  enterprise  &  sponsored  its  completion 

-  The  purpose  of  the  standard  is  to  provide  a  common  framework  for  sharing 
information  throughout  the  VV&A  processes 

-  The  common  method  of  documentation  benefits  participants  in  the  VV&A 
processes  by  eliminating  unnecessary  redundancy  and  facilitating  reuse  of 
information  when  accrediting  an  M&S  for  an  intended  use 

-  Was  published  in  January  2008  &  has  received  positive  feedback  from 
users 

-  Enables  the  efficient  reuse  of  VV&A  information  to  include  discoverability, 
accessibility,  &  usability 

-  Provides  a  common  way  of  documenting  information  about  VV&A  of  M&S 

-  Allows  information  about  VV&A  projects  to  be  machine  searchable  thereby 
promoting  reuse 
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Acquisition  Commitment  to  VV&A 
Sponsored  Funding  (cont.) 


*  To  finalize  the  DoD  VV&A  Documentation  Tool: 

-  Leveraged  efforts  started  by  the  Navy 

-  Automated  production  of  the  MIL-STD  3022  templates 

-  Currently  20  customers  are  using  the  tool 

-  The  DoD  VV&A  Documentation  Tool  can  be  accessed  through  the 
M&S  Coordination  Office  website: 

http://www.msco.mil/vva  doc  tool.html 

-  Currently  requires  a  Common  Access  Card  (CAC)  or  External 
Certification  Authority  (ECA) 

-  We  see  a  commercial  documentation  standard  under  SISO  as  a 
logical  next  step 
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Acquisition  Commitment  to  VV&A: 
Sponsored  Funding  (cont.) 


*  To  establish  a  risk-based  approach  for  VV&A 

-  Since  there  is  a  cost  for  verifying  &  validating  models  &  simulations,  the 
effort  is  intended  to  help  a  M&S  user  determine  how  to  focus  the  V&V  to  get 
the  information  needed  to  accredit  the  M&S  for  use. 

-  The  importance  of  VV&A  is  directly  related  to  the  criticality  of  the  decision 
being  informed  by  M&S. 

-  This  effort  started  in  the  spring  of  2009  and  is  scheduled  to  be  completed  in 
the  spring  of  2011 

-  A  commercial  standard  under  SISO  may  follow 

•  To  develop  a  Capability  Maturity  Model  Integration  (CMMI)- 
like  VV&A  maturity  model 

-  The  primary  focus  of  this  task  will  be  the  establishment  of  a  CMMI-like 
maturity  model  to  support  a  clearer  articulation  of  the  level  of  VV&A  that  an 
organization  can  or  should  achieve. 

-  Task  includes  the  development  of  a  VV&A  roadmap  that  defines  the  gaps 
that  inhibit  efficient  and  effective  VV&A  implementation 

-  M&S  Steering  Committee  approved  funding  for  FY-10  &  FY-1 1 

-  A  statement  of  work  has  been  submitted  with  an  anticipated  work  start  by 
December  2009 
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VV&A  Guidance  Efforts 


*  Strengthened  the  Defense  Acquisition  Program  Support  (DAPS) 
Methodology  for  Program  Support  Reviews  (PSRs)  by  identifying 
VV&A  as  a  key  aspect  of  a  program’s  M&S  strategy. 

*  Developed  and  provided  a  M&S  tutorial  to  Assessments  and  Support 
that  emphasized  the  importance  of  VV&A  documentation. 

*  Addressed  VV&A  in  multiple  college-level  courses  developed  under 
the  “M&S  Education  for  the  Workforce”  Project 

*  Addressed  VV&A  in  two  Defense  Acquisition  University  Continuous 
Learning  Modules: 

-  M&S  for  Systems  Engineering 

-  M&S  for  Test  &  Evaluation 

*  Addressed  VV&A  in  the  “M&S  Guidance  for  the  Acquisition 
Workforce”  and  the  Defense  Acquisition  Guidebook  (DAG)  Chapter 
4.5.8  or  go  to: 

http://www.aca.osd.mil/sse/docs/M-S-Guidance-Acauisition-Workforce.pdf 
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AMSWG  VV&A  Subcommittee 


*  The  Chair  of  the  Acquisition  M&S  Working  Group 
(AMSWG)  proposed  and  established  the  VV&A 
Subcommittee  in  July  2009. 

-  Acquisition  is  the  biggest  producer  and  consumer  of  VV&A  products 

-  Congress  has  expressed  interest  in  metrics  with  respect  to  the  number  of 
models  and  simulations  with  documented  VV&A 

-  Periodic  VV&A  technical  interchange  meetings  are  needed  to: 

-  promote  the  importance  of  VV&A 

-  exchange  VV&A  information 

-  make  VV&A  information  available 

-  provide  a  virtual  VV&A  brain  trust-capability  for  the  Acquisition  Community 

-  respond  to  requests  for  VV&A  information 

-  Supports  Action  4-5  in  the  AMSMP  and  Goal  3  of  the  M&S  Steering 
Committee’s  Strategic  Vision  for  DoD  M&S 
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Background  on  DoDI  5000.61 


•  DoDI  5000.61  DoD  Modeling  and  Simulation  (M&S)  Verification, 
Validation,  and  Accreditation  (VV&A)  is  the  Department’s  instruction 
that  implements  policy,  assigns  responsibilities,  &  prescribes 
procedures  for  the  verification,  validation,  &  accreditation  (VV&A)  of 
DoD  models,  simulations,  &  associated  data. 

•  The  latest  version  of  the  DoDI  5000.61  was  last  dated  May  13, 2003 

•  At  the  July  2007  M&S  Steering  Committee  Offsite,  the  M&S  SC 
directed  a  review  and  update  if  needed  to  the  DoDI  5000.61 

•  Office  of  the  Secretary  of  Defense  Program  Analysis  &  Evaluation 
(PA&E),  a  member  of  the  Modeling  &  Simulation  Steering  Committee, 
volunteered  to  lead  the  effort  to  revise  the  instruction  in  3  months. 

•  Their  effort  ended  when  the  revised  instruction  began  formal  staffing 
for  approval  in  June  2009 
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DoDI  5000.61 :  Current  Status 


*  The  instruction  entered  formal  SD  106  staffing  on 
June  4,  2009 

*  Suspense  for  formal  comments  was  August  1 1 ,  2009 

*  The  M&S  Coordination  Office  is  led  the  adjudication 
process 

*  Adjudication  is  nearly  complete 

*  USD  (AT&L)  Signature  is  expected  by  the  end  of  this 
calendar  year 
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NATO  Participation 


*  NATO  has  requested  more  U.S.  participation  in 
M&S  activities 

*  The  Modeling  &  Simulation  Coordination  Office 
has  requested  more  participation  from  the 
Communities  and  Services 

*  The  Acquisition  Community  has  volunteered  to 
participate  in  NATO  VV& A  activities  and  others 

*  Coordination  with  the  M&S  CO  is  underway 
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Summary 


*  The  Acquisition  M&S  Community  is  committed  to 
VV&A 

*  The  Acquisition  M&S  Community  wants  to  further 
mature  the  practice  of  VV&A 

*  We  solicit  your  support  and  participation 

*  Let  us  know  your  concerns  and  suggestions 

*  We  will  incorporate  and  champion  good  ideas 

Michael  Truelove 
ODDR&E/Systems  Engineering 
703-412-3683 

Michael.Truelove@syseng-so.com 
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Q  &  A 
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Acronyms 


•  AMSMP:  Acquisition  Modeling  &  Simulation  Master  Plan 

•  AMSWG:  Acquisition  Modeling  &  Simulation  Working  Group 

•  AP  EXCOM:  Adaptive  Planning  Executive  Committee 

•  AT&L:  Acquisition,  Technology  and  Logistics 

•  CAC:  Common  Access  Card 

•  CAPE  &  JS:  Cost  Assessment  Program  Evaluation  &  Joint  Staff 

•  CMMI:  Capability  Maturity  Model  Integration 

•  COCOMS:  Combatant  Commands 

•  DAPS:  Defense  Acquisition  Program  Support 

•  DDR&E:  Director,  Defense  Research  and  Engineering 

•  DIMSCG:  Defense  Intelligence  M&S  Collaboration  Group 

•  DoD:  Department  of  Defense 

•  DoDI:  Department  of  Defense  Instruction 

•  DOT&E:  Director  Operational  Test  and  Evaluation 

•  ECA:  External  Certification  Authority 

•  INCOSE:  International  Council  on  Systems  Engineering 

•  JADM:  Joint  Analytic  Data  Management 

•  JCDE  EC:  Joint  Concept  Development  &  Experimentation  Executive  Committee 
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Acronyms  (cont.) 


•  JFCOM:  Joint  Forces  Command 

•  M&S:  Modeling  and  Simulation 

•  MA:  Mission  Assurance 

•  MBSE:  Model  Based  Systems  Engineering 

•  MIL-STD:  Military  Standard 

•  NDIA:  National  Defense  Industrial  Association 

•  ODDR&E:  Office  of  the  Director,  Defense  Research  and  Engineerint 

•  OSD:  Office  of  the  Secretary  of  Defense 

•  OUSD:  Office  of  the  Under  Secretary  of  Defense 

•  P&R:  Personnel  and  Readiness 

•  PSR:  Program  Support  Review 

•  SE:  Systems  Engineering 

•  SISO:  Simulation  Interoperability  Standards  Organization 

•  SLC:  System  Life  Cycle 

•  T2  ESG:  Training  /  Transformation  Executive  Steering  Group 

•  USD(I):  Under  Secretary  of  Defense  for  Intelligence 

•  VV&A:  Verification,  Validation  and  Accreditation 

•  WG:  Working  Group 
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Standards  Needs  Identified  in  the 
Acquisition  M&S  Master  Plan 


•  Action  1-5  Contracting  Guidelines  /  Best  Practices 

•  Action  2-2  Commercial  Systems  Engineering  (SE) 
Standards 

•  Action  2-3  Distributed  Simulation  Standards 

•  Action  2-4  DoD  Architecture  Framework  (DoDAF) 

•  Action  2-5  Metadata  Template  for  Reusable  Resources 

•  Action  3-2  Best  Practices  for  M&S  Development 

•  Action  3-3  Distributed  Live,  Virtual,  Constructive  (LVC) 
Environments 

•  Action  4-2  M&S  Planning  Best  Practices 

•  Action  4-5  VV&A 

•  Action  5-1  Required  M&S  Competencies 

•  Action  5-3  M&S  Body  of  Knowledge 
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Disciplined  Systems  Engineering 


& 

The  Foundational  Team 
SE  Skill  Set 


Charles  M.  Garland 

Air  Force  Center  for  Systems  Engineering 

(937)  255-3355  x3368 
Charles.Garland@afit.edu 


October  26,  2009 


The  Issue 


Develop  America's  Airmen  Today ...  for  Tomorrow 


>  Chief  Engineer’s  are  accountable  for  implementing  a  disciplined 
systems  engineering  process  with  the  skills  of  the  staff  they  have 


>  Staff  planning  is  required  content  in  the  Systems  Engineering  Plan 


How  does  the  chief  engineer  know  that  the  skills  of  their  staff  are 
adequate  to  implement  a  disciplined  systems  engineering  process? 


What  to  Do 


Develop  America's  Airmen  Today ...  for  Tomorrow 


>  Define  the  minimum  collective  staff  skill  set  required  to  implement 
a  disciplined  systems  engineering  process 


>  Create  an  assessment  tool  for  the  Chief  Engineer 

>  Identify  skills  of  the  current  staff 

>  Identify  gaps  between  current  staff  skills  and  the  defined 
minimum  collective  skill  set 


>  Publish  the  minimum  collective  staff  skill  set  and  assessment  tool 
in  the  appropriate  Air  Force  level  document. 


Outline  of  Basic  Tasks 


Develop  America's  Airmen  Today ...  for  Tomorrow 


>  Establish  the  process 

>  Develop  the  skills  dictionary 

>  Assessment  tool  selection 

>  Pilot  project,  wing  level 

>  Draft  policy  guidance 

>  Publish  policy  guidance 


Establish  the  Process 


Develop  America's  Airmen  Today ...  for  Tomorrow 


>  Who  owns  the  process? 

>  Wing/Group/Squadron  Chief  Engineer? 

>  Process  covers  the  technical  staff 

>  Process  should  not  require  additional  manpower 


Develop  the  Skills  Dictionary 

Develop  America's  Airmen  Today ...  for  Tomorrow  “ 

>  Skills  should  cover  SE  process  execution  across  the  life  cycle 

>  Not  intended  to  cover  detailed  skills  of  experts  in  a  specific 
technical  discipline 

>  Skill  set  should  be  adaptable  and  capable  of  easy  modification 


Team  has  identified  32  skills  within  the  skills  dictionary  labeled 
“Foundational  Team  SE  Skill  Set”. 


Competency  Model 


Develop  America's  Airmen  Today ...  for  Tomorrow 


Systems  Engineering  Foundational 
Team  Competency  Model 


Foundational  Team  SE  Skill  Set 
V  Slide  1 

Develop  America's  Airmen  Today ...  for  Tomorrow  m 

Decision  Analysis 

•Can  carefully  identify  the  decision  at  hand  and  the  decision  context  using  appropriate  disciplines  and  disciplined  practices 
•Can  articulate  one’s  objectives  in  the  situation  at  hand 
•Can  articulate  the  benefits  of  decision  analysis 

•Can  identify  the  various  elements  of  the  situation,  i.e.  1)  values,  priorities  and  objectives,  2)  decisions  to  be  made,  3) 

certain  and  uncertain  events,  4)  consequences,  5)  assumptions,  6)  limitations  and  constraints,  7)  stakeholders,  8)  variation 

•Can  structure  decisions  using  influence  diagrams  and/or  decision  trees  and/or  other  methods  and  models 

•Can  structure  values  and  objectives 

•Can  specify  ways  to  measure  achievement  of  objectives 

•Can  apply  a  decision-analysis  process 

•Can  assess  the  completeness,  fidelity,  suitability  and  implications  of  the  decision  context 

Technical  Planning 

•Can  plan  and  properly  sequence  all  technical  steps 
•Can  manage  risks  from  out-of-sequence  technical  steps 

•Can  manage  the  application  of  all  relevant  technical  specialties  at  the  right  time 
•Can  link  the  technical  process  with  business  and  management  processes 
•Can  plan  and  properly  sequence  all  technical  review  activities 
•Can  plan  and  manage  testing  as  well  as  validation  and  verification  activities 
•Can  plan  and  manage  technical  activities  that  span  the  acquisition  phases 

•Can  develop,  tailor,  implement,  integrate  and  manage  the  following  technical  processes:  requirements  development, 
logical  analysis,  design  solution,  implementation,  integration,  verification,  validation,  and  transition. 


Foundational  Team  SE  Skill  Set 
^  Slide  2 

Develop  America's  Airmen  Today ...  for  Tomorrow 

Risk  Management 

•Can  actively  manage  risk  on  a  continuing  basis  across  the  life  cycle 
•Can  adjust  risk  handling  plans  to  changing  circumstances 

•Can  identify  potential  additional  risks  resulting  from  interactions  between  multiple  risk  sources 

Configuration  Management 

•Can  implement  control  processes  that  regulate  change  at  all  levels  of  the  product  hierarchy  from  the  system-of-systems 
level  down  to  the  piece  part  level 

Technical  Data  Management 

•Can  implement  control  processes  that  regulate  the  creation,  change,  dissemination,  archiving  and  retrieval  of  technical 
data 

Interface  Management 

•Can  implement  control  processes  that  regulate  the  creation  of,  change  to,  and  agreement  with  documented  descriptions 
of  the  physical  and  functional  boundaries  at  all  levels  of  the  product  hierarchy. 

•Can  establish  forums  for  interface  management  when  no  single  organization  has  responsibility  for  the  entire  product 
hierarchy. 


Foundational  Team  SE  Skill  Set 
^  Slide  3 

Develop  America's  Airmen  Today ...  for  Tomorrow 

Requirements  Management 

•Can  establish  control  processes  that  regulate  the  establishment  of,  change  to,  and  agreement  with  a  specific  documented 
set  of  technical  requirements  at  a  given  level  of  the  product  hierarchy. 

•Can  articulate  the  effect  of  user  requested  changes  on  documented  technical  requirements  at  all  levels  of  the  product 
hierarchy. 

•Can  identify  risks  of  achieving  specific  technical  requirements  for  input  into  the  risk  management  process. 

•Can  complete  trade  studies  that  balance  an  achievable  set  of  requirements  within  cost  and  schedule  constraints. 

Technical  Assessment 

•Can  establish  and  use  technical  performance  measures  to  track  achievement  of  key  performance  parameters 
•Can  establish  and  use  leading  indicators  to  monitor  the  health  and  status  of  systems  engineering  process  execution 
•Can  conduct  thorough,  comprehensive  technical  reviews  that  accurately  convey  technical  progress 
•Can  use  manufacturing  and  technology  readiness  levels  to  judge  the  risk  of  achieving  desired  technical  requirements 


Tool  Selection 


Develop  America's  Airmen  Today ...  for  Tomorrow 


>Tool  should  be  easily  accessible  to  chief  engineers 

>Tool  should  allow  easy  skill  set  adjustments 

>Tool  should  be  common  across  wing/group/squadron 

>Tool  will  be  an  individual  self  assessment  for  all  32  skills  identified 

>  Proficiency  level 

>  Importance  -  aid  to  judging  if  we  have  the  correct  skill  mix 
>  Supervisor  review  of  individual  self  assessments 
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Develop  America's  Airmen  Today ...  for  Tomorrow 


Self  Assessment 
Proficiency  Level 


>  No  Skill  -  neither  understands  concepts  of  nor  can  apply  this  skill 

>  Newbie  -  general  understanding  of  concepts  and  can  apply  skill 
under  supervision 

>  Capable  -  general  understanding  of  concepts  and  can  apply  skill 
with  guidance 

>  Proficient  -  depth  of  understanding  of  concepts  permits 
independent  application 

>  Expert  -  leads  others  in  applying  this  skill  and  is  sought  out  by 
others  for  guidance 


Self  Assessment 
Importance 


Develop  America's  Airmen  Today ...  for  Tomorrow 


>  Not  Important  -  this  skill  is  not  important  for  success 

>  Useful  -  this  skill  is  helpful  but  not  necessary  for  success 

>  Important  -  having  this  skill  contributes  to  success 

>  Vital  -  it  is  not  possible  to  be  successful  without  this  skill 


w 


Assessment  Tool  Format 


Develop  America's  Airmen  Today ...  for  Tomorrow 


Technical  Assessment 


Skill  1  -  Can  establish  and  use  technical  performance  measures  to  track  achievement  of  key  performance 
parameters. 


Importance 

Proficiency  Level 

Not  Important  -  this  skill  is  not  important  for  success 

Not  Important 

No  Skill 

Useful  -  this  skill  is  helpful  but  not  necessary  for  success 

Useful 

Newbie 

Important  -  having  this  skill  contributes  to  success 

Important 

Capable 

Vital  -  it  is  not  possible  to  be  successful  without  this  skill 

Vital 

Proficient 

Expert 

No  Skill  -  neither  understands  concepts  of  nor  can  apply  this  skill 

Skill  2  -  Can  establish  and  use  leading  indicators  to  monitor  the  health  and  status  of  systems  engineering 


Newbie  -  general  understanding  of  concepts  and  can  apply  skill  under  supervision 


Importance 

Proficiency  Level 

Proficient  ■ 

-  depth  of  understanding  of  concepts  permits  independent  application 

Not  Important 

No  Skill 

Expert  -  leads  others  in  applying  this  skill  and  is  sought  out  by  others  for  guidance 

Useful 

Newbie 

Important 

Capable 

Vital 

Proficient 

Expert 

Skill  3  -  Can  conduct  thorough,  comprehensive  technical  reviews  that  accurately  convey  technical  progress. 


Importance 

Proficiency  Level 

Not  Important 

No  Skill 

Useful 

Newbie 

Important 

Capable 

Vital 

Proficient 

Expert 

Skill  4  -  Can  use  manufacturing  and  technology  readiness  levels  to  judge  the  risk  of  achieving  desired 
technical  requirements. 


Importance 

Proficiency  Level 

Not  Important 

No  Skill 

Useful 

Newbie 

Important 

Capable 

Vital 

Proficient 

Expert 

Pilot  Project,  Wing  Level 


Develop  America's  Airmen  Today ...  for  Tomorrow 


>  Analogous  to  “fly-fix-fly”  approach  in  aircraft  acquisition 

>  Learn  as  you  go  and  refine  the  bumps  in  the  road  before  issuing 
policy  guidance 

>  Seek  willingness  to  participate  at  the  wing  level 


303rd  AESW  will  participate  in  the  pilot  project. 


Draft  Policy  Guidance 

Develop  America's  Airmen  Today ...  for  Tomorrow 

>  Coordinate  draft  within  the  team 

>  Refine  the  draft  with  lessons  learned  from  the  pilot  program 

>  Gain  consensus  to  proceed  with  formal  publication 
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Protect  the  Mission , 
Promote  Growth, 
and  Preserve  Legacy 


Presented  by:  Patricia  Scaramuzzo 


©  2009  Lockheed  Martin,  All  Rights  Reserved 


Challenge 


What  did  the  Knowledge  Continuity*  project  seek  to  do? 
FIND  EFFECTIVE  KNOWLEDGE  TRANFER 

-  “The  problems  with  transferring  deep  smarts  are  many.  ..you 
often  don't  know  what  you  know  or  bring  it  into  conscious 
consideration  until  you  are  forced  to  explain  or  demonstrate  it  in 
response  to  some  specific  situation.  ” 

-  Source:  ASK  Magazine  volume  22,  Dorothy  Leonard 

-  "In  the  KM  world,  a  Holy  Grail  is  effective  capture, 
enhancement,  persistence,  and  transfer  of  knowledge." 

-  Source: 
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*  Patent  Pending 


Approach 


Get  “academically  smart”  on  Knowledge  Transfer  -  some  examples: 

Deep  Smarts:  Dorothy  Leonard 
Lost  Knowledge:  David  DeLong 
ASK  Magazine:  NASA 

Mine  best  practices  across  the  corporation 

Identify  unique  approaches  in  Knowledge  Sharing 
Drill  it  down  into  a  process 

Collaboratively  develop  a  new  Knowledge  Transfer  approach 
Pilot,  and  study  results 
Expand  the  practice 
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Timeline 


2003-2005  1 Q-2Q2008  1 Q-2Q2009 

Organic  Consolidation 

Development  of  Programs  Refinement 


Socialization  Pilot  Surge 

2005-2007  3Q2008-2Q2009  2009 
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Results  of  the 

Knowledge  Continuity  Pilot 
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Aggregate  of  Pilot  Team  data 
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0%  20%  40%  60%  80%  100% 


New  ramp-up 


Successful  Pilot  Teams 
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Pilot  Teams  that  Struggled 
. .  .didn  ’t follow  processes 
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KC  Surge  (post-Pilot)  since  1/1/09  7^ 


Know!  ledge 


Know)  [edge 


Knowj  [edge 


Protect  Promote  Preserve 
I  Mission  *  Growth  *  Legacy 


Knowj  edge 


Protect  Promote  Preserve 
I  Mission  *  Growth  Legacy 


Know)  [edge 


Protect  Promote  Preserve 
I  Mission  *  Growth  *  Legacy 


Know)  [edge 


Protect  Promote  Preserve 
I  Mission  *  Growth  *  Legacy 


Know)  [edge 


■  Across  every  Business  Area  in  LMC!! 

■  Growth  has  mostly  been  organic 
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Growth:  Knowledge  Continuity 


ZOOB  Pilot 

2009  Teams 

2009  Teams 

Teams 

YTD 

Projected 

Continuity 


Knowi  edge 


Continuity 


Continuity 


Know  edae 


Continuity 


Continuity 


Knowj  edge 


Continuity 


Continuity 


Knowj  ledge 


Continuity 


Know)  edge 


Continuity 


Know  edae 


Knowi  edge 


Continuity 


Know  edae 


Knowj  edge 


Continuity 


Knowj  edge 


Continuity 


Continuity 


Continuity 


Continuity 


KC  Surge  (post-Pilot)  since  1/1/09  7^ 


■  KC  exists  in  every  Business  Area  across  LMC 

-  -400  individuals,  29  facilitators  trained  in  LMC  KC  process  to-date 

-  56  teams  since  2008  have  employed  KC  to  capture  at-risk  knowledge 
critical  to  business/programs 

-  1 35%  higher  KC  participation  since  2008  Pilot,  will  be  much  larger  if 
expectations  are  realized 

-  KC  used  on  key  defense  programs  to  protect  critical  defense 
knowledge  &  LMC  Program  Performance  Management  /  Earned  Value 

-  Forbidden 
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KC  Cases:  _ 

Stories  on  Knowledge  Continuity  Story  Telling 

Retirement 

•  Knowledge  of  mission  s/w 

•  Individual  needing  the 
knowledge  had  1 5  yrs 
experience  in  flight  s/w 


This  SSI  KC  Pilot  team: 

•  Captured  artifacts  on  classified  network 

•  Knowledge  used!  Monte  Carlo  analysis 

•  Found  leadership  extremely  supportive 

•  Expended  only  60  hours  &  now  has  a  new  Expert  up-to-speed 
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W/Jzn  zyzryo/m  on  u  KC  Tznm  w  Engiigsd, 

'C)  M/  ^  /  o  r  ~  fV. 
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1 00%  of  Expert  s  role  was  transferred 

Individual  needing  the  knowledge  feels 
“up-to-speed,”  “more  engaged,” 
“gaining  expertise” 


KC  Cases:  _ 

Stories  on  Knowledge  Continuity  Story  Telling 


Deployable,  Exponential  Growth 


•  Expert  close  to  retirement 
(32  yrs) 


•  3  individuals  rec’d  expertise 


•  BA  needed  skills  growth 


Two  individuals  assigned  to  another  BA 
site 

Remaining  individual  stayed  on  program 


This  SSI  KC  Pilot  team: 

•  Tailored  KC  process 

•  Captured  all  topics  identified  on  SharePoint  folder  &  Unity 

•  Expended  only  72  hours 
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KC  Cases :  _ 

Stories  on  Knowledge  Continuity  Story  Telling 


Unity  Artifacts... Useful  Application 


Expert  was  more  junior  but 
had  unique  expertise 


Others  ranged  from  highly 
experienced  to  inexperienced 


Developed  a  template  for  Unity 
space  for  future  KC  teams 

Newer  employees  feel  “up-to- 
speed,”  “more  engaged” 


This  SSI  KC  Pilot  team: 

•  Tailored  the  KC  process/artifacts  on  Unity! 

•  During  KC,  worked  on  collaboration  projects 

•  Saved  $  using  knowledge  transferred  to  develop 
Ambassador's  Guide 
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♦ 


Conclusions 


■  Successful  Knowledge  Transfer  so  that  knowledge  is  pervasive  is  possible 

-  Ease  retirements  and  reduce  single  points  of  failure 

-  Exponentially  grow  experts;  assure  performance 

-  Speed  ramp  up 

■  Critical  to  success  is  piloting  processes  and  techniques  to  find  what  works 

-  Multi-talented,  multi-disciplinary,  multi-business  team  was  key 

-  Leadership  support  most  critical 

-  Recognize  expertise  as  non-generational 

-  Less  spoon  feeding,  more  empowering  and  enabling 

-  Make  “bubble  up”  of  KC  projects  as  easy  and  possible  as  “top  down”  encouragement 

-  Don’t  just  ask  an  expert  to  “share  their  knowledge”  (e.g.,  ad-hoc  mentoring  model) 

-  Provide  techniques  and  processes  to  assure  success 

-  Show  them  the  What’s  in  it  For  Me  (WIIFM) 
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Presentation  Outline 


■  Study  Background 

■  Study  Objectives  and  Approach 

■  Surveys  of  M&S  Tool  Managers  and  Users 

■  Categories  of  Tool  Management  Approaches 

■  Taxonomy  for  Assessing  Success  of  Management  Approaches 

■  Preliminary  Assessment  of  Success  Attributes  for  M&S  Tool 
Management 

■  Future  Work 
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Management  Concepts  for  Broadly  Needed  M&S  Tools 
Study  Background  (1  of  2) 

■  Certain  M&S  tools  are  common  to  multiple  programs  and 
organizations 

■  Many  government-managed  models  and  simulations  are  already 
used  broadly 

■  However,  such  broadly-used  M&S  tools  typically  suffer  from 
several  problems,  including 

■  A  lack  of  adequate  model  manager  funding,  and 

■  A  stakeholder  requirements  management  council  to: 

■  allow  the  incorporation  of  tool  enhancements  developed  by  users 
into  the  standard  version  (“street  version”), 

■  improve  the  model’s  accuracy  by  examining  discrepancies 
between  the  model  and  actual  test  results  (the  “fix”  step  of  the 
“model-test-fix-model”  process),  and 

-  build  in  new  capabilities  to  meet  foreseeable  needs,  such  that  the 
capabilities  can  be  delivered  by  the  time  users  need  them 
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Management  Concepts  for  Broadly  Needed  M&S  Tools 
Study  Background  (2  of  2) 

■  Study  is  sponsored  by  the  Director,  Office  of  the  Director  of 
Systems  and  Software  Engineering  (D,  SSE)  in  the  Office  of  the 
Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  (OUSD(AT&L)) 

■  On  behalf  of  the  Acquisition  M&S  Working  Group  (AMSWG) 

■  Study  is  an  initial  step  in  addressing  Acquisition  M&S  Master 
Plan  (AMSMP)  Action  3-4  (“Centrally  fund  and  manage  the 
development  of  high-priority,  broadly-needed  M&S  tools”) 

■  Before  embarking  on  such  an  initiative,  it  is  prudent  to 
objectively  study  DoD’s  current  experience  in  the  management 
of  broadly-needed  tools 

■  Attempt  to  identify  innovative  approaches  that  could  be  leveraged 
to  improve  the  cost-effectiveness  of  DoD  M&S  tools  more  broadly 
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Management  Concepts  for  Broadly  Needed  M&S  Tools 
Study  Objectives 

■  Identify  best  practices  for  managing  broadly-needed 
M&S  tools 

■  Based  on  these  findings,  recommend  actions  the 
U.S.  DoD  should  take  to  improve  its  management  of 
such  M&S  tools 


Management  Concepts  for  Broadly  Needed  M&S  Tools 
Study  Approach 


"  Develop  list  of  M&S  tools  used  by  multiple  organizations 
not  under  the  same  chain  of  command  or  contract 

■  Survey  M&S  tool  managers  and  users  on  management 
approaches 

■  Document  and  categorize  management  approaches  for 
the  tools  identified 

■  Assess  degree  of  success  each  tool  management 
approach  has  had  in  avoiding  certain  problems 

■  Develop  a  taxonomy  for  assessing  success  of  M&S  tool 
management  approaches 

-  Identify/develop  best  practices  for  managing  broadly 
needed  M&S  tools 

■  Recommend  actions  DoD  should  take  to  improve  its 
management  of  broadly-needed  M&S  tools 

-  Develop  list  of  desirable  characteristics  of  candidate 
tools  to  be  used  in  pilot  applications 


Done  -  but  still 
growing 

Done  -  but  still 
accepting  inputs 

Done  -  but  open 
to  update 

Done,  based  on 
survey  responses 

Taxonomy 

developed 

Success  attributes 
developed 

In  progress 


List  of  M&S  Tools  with  Responses  to  Tool  Manager  Survey 
(31  responses  on  27  tools) 


•  Advanced  Joint  Effectiveness  Model  (AJEM) 

•  Advanced  Testing  Capability  (ATC) 

•  Battle  Command  Management  Service 
(BCMS) 

•  Comprehensive  Mine  and  Sensor  Simulator 

•  Extended  Air  Defense  Simulation  (EADSIM) 

•  Hazard  Prediction  and  Assessment  Capability 
(HPAC) 

•  Intelligence  Modeling  and  Simulation  for 
Evaluation 

•  Joint  Analysis  System  (JAS) 

•  Joint  Conflict  and  Tactical  Simulation  (JCATS) 

•  Joint  Communication  Simulation  System 
(JCSS) 

•  Joint  Integrated  Mission  Model  (JIMM) 

•  Joint  Semi-Automated  Forces  (JSAF)  (JFCOM 
version) 

•  Joint  Theater  Level  Simulation  (JTLS) 


•  Langley  Standard  Real-Time  Simulation 
in  C++  (LaSRS++) 

•  Model  for  Intratheater  Deployment  by 
Air/Sea  (MIDAS) 

•  Naval  Simulation  System  (NSS) 

•  One  Semi-Automated  Forces  (OneSAF) 

•  OpenEaagles  Simulation  Framework 

•  ProtoCore 

•  Role  Player  Workstation 

•  RunTime  Infrastructure  (RTI)  -  MATREX 

•  RTI  NG  Pro 

•  Simulation  Display  (SIMDIS) 

•  SPIRITS 

•  Suppressor 

•  Synthetic  Theater  Operations  Research 
Model  (STORM) 

•  Threat  Modeling  and  Analysis  Program 
(TMAP) 

- dPL 


Questions  on  the  M&S  Tool  User  Survey 

Responder  Information 

I)  Name  2)  Rank/Title  3)  Organization  4)  Email  Address  5)  Phone  Number 

Requirements  Management 

6)  How  should  user  requirements  be  prioritized  when  funding  and/or  schedule  are  insufficient  to  meet  all  requirements? 

Configuration  Management 

7)  Is  it  critical  to  maintain  a  single  source  baseline,  or  are  there  circumstances  under  which  multiple  forks  should  be 
permissible?  What  criteria  should  be  used  to  make  this  decision? 

8)  Identify  good  tool  distribution  mechanisms/methods  (for  source,  executable,  or  both). 

9)  How  frequent  should  releases  be?  Please  describe  the  criteria  upon  which  the  frequency  may  depend,  e.g. 
tool  maturity,  criticality  of  bug  fixes. 

Code  Development 

10)  Should  externally  developed  code  (by  users  or  others)  be  integrated  into  the  code  baseline? 

II)  How  should  conflicts  between  modifications  submitted  by  different  users/co-developers  be  mediated? 

Test  Management 

12)  Should  V&V  be  a  formal  part  of  the  integration  process? 

13)  What  processes/products  are  critical  prior  to  product  release,  e.g.,  regression  testing,  reference  data? 

Lessons  Learned 

14)  Please  describe  any  other  management  best  practices  that  are  critical  to  successful  model  management. 
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Categories  of  Tool  Management  Approaches  (1  of  2) 


■  Government  Coordinated  (GC) 

■  A  single  government  office  coordinates  development  of  one 
version  of  the  tool  for  all  users.  Government  mechanisms,  like 
MIPRs,  are  used  to  contribute  funds.  Developers  (contractors  or 
DoD  employees)  are  paid  and/or  directed  through  a  single 
coordinator. 

■  Developer  Coordinated  (DC) 

■  A  single  development  contractor  coordinates  one  version  of  the 
tool  for  all  users.  Commercial  mechanisms,  like  license  fees  or 
development  contracts,  are  used  to  contribute  funds  from  users. 

■  Independent  Development  (ID) 

■  One  or  more  developers  (contractors  or  DoD  employees)  produce 
their  own  versions  from  a  common  tool  baseline.  Each  user  is  free 
to  select  a  version  and/or  developer. 


Categories  of  Tool  Management  Approaches  (2  of  2) 


■  Government  Open  Source  Hybrid  (GOSH) 

■  A  government  office  authorizes  certain  developers  (contractors  or 
DoD  employees)  to  participate  in  a  shared  source  effort.  Each  user 
chooses  a  developer  and  all  changes  are  constantly  available  to  all 
participants. 

■  Open  Source  (OS) 

■  One  or  more  developers  (contractors  or  DoD  employees) 
participate  in  a  shared  source  baseline.  Each  user  chooses  a 
version  to  use.  No  contractual  relationship  necessarily  exists 
between  users  and  developers. 

■  Independent  “Co-opetition”  (1C) 

■  One  or  more  developers  (contractors  or  DoD  employees)  produce 
independent  changes  to  a  shared  baseline.  Each  user  chooses  a 
developer,  and  the  user  determines  if  and  when  their  changes  are 
made  available  for  inclusion  in  future  baselines. 

MiUmi _ /jDI 

ilirfiV  nrL- 


Taxonomy  for  Judging  Success  of  Approaches  - 
Meeting  Foreseeable  Needs 


High  -  manager  solicits  inputs  to  future  needs;  manager 
prioritizes  requirements  and  integration  activities  to  meet 
projected  user  community  needs 

■  Medium  -  priorities  are  set  by  a  configuration  control  board; 
users  may  provide  additional  funding  to  meet  their  specific 
requirements 

■  Low  -  projected  user  community  needs  are  not  considered  in 
the  requirements  and  integration  process 


Taxonomy  for  Judging  Success  of  Approaches  - 
Integrating  User-Developed  Enhancements 


High  -  manager  has  structured,  documented  process  for 
evaluating  user  enhancements  and  integrating  them  into  the 
standard  version;  process  includes  regression  testing  and 
mediation  of  differences  between  submitted  changes 

■  Medium  -  enhancements  from  a  recognized  set  of  sources  are 
accepted  and/or  the  framework  allows  for  users  to  individually 
integrate  their  own  plug-ins  or  libraries 

■  Low  -  integration  of  user-developed  enhancements  is  on  an  ad 
hoc  basis  or  not  at  all 


Taxonomy  for  Judging  Success  of  Approaches  - 
Model  Accuracy  (Verification  and  Validation) 


High  -  validation  or  testing  of  the  fully  integrated  tool  is 
required  as  part  of  the  structured  management  process 

■  Medium  -  manager  accepts  validation  data  where  available,  but 
does  not  require  it 

■  Low  -  management  process  does  not  include  V&V 


13 


Taxonomy  for  Judging  Success  of  Approaches  - 
Customer  Support 


High  -  manager  provides  broad  and  responsive  customer 
support  including  live  support  (help  desk)  and  extensive 
documentation  that  supports  understanding  and  use  of  the 
model;  manager  actively  communicates  with  user  community 

■  Medium  -  manager  provides  documentation  beyond  just 
technical/user’s  manual  and  live  support 

■  Low  -  manager  provides  technical/user’s  manual;  live  support 
is  on  an  ad  hoc  basis 


M&S  Tool  Management  Success  Attributes  (1  of  3) 


M&S  Tool  Management  Success  Attributes: 

"The  M&S  Tool  Manager  ...” 

Meeting 

Foreseeable 

Needs 

Integrating 

User-Developed 

Enhancements 

Model 

Accuracy 

(V&V) 

Customer 

Support 

1 .  Successfully  solicits  recommendations 
from  users  for  new  capabilities. 

X 

2.  Has  a  process  for  managing  the  tool 
baseline(s)  that  prevents  irreconcilable 
divergence. 

X 

X 

3.  Has  implemented  into  the  baseline  tool 
enhancements  agreed  upon  by  a  peer  / 
user  review  process. 

X 

X 

4.  Provides  /  publishes  justification  for  not 
including  any  suggested  tool 
enhancements  that  were  not  included  in 
the  new  baseline  tool. 

X 

X 

X 

5.  Actively  communicates  with,  and 

engages,  users  /  external  developers  on 
a  consistent  basis  concerning  tool 
efficacy  and  applicability. 

X 

X 

- dPL 


M&S  Tool  Management  Success  Attributes  (2  of  3) 


M&S  Tool  Management  Success  Attributes: 

"The  M&S  Tool  Manager  ..." 

Meeting 

Foreseeable 

Needs 

Integrating 

User-Developed 

Enhancements 

Model 

Accuracy 

(V&V) 

Customer 

Support 

6.  Has  implemented  a  process  to  acquire 
and  assess  (using  a  peer  /  user  review 
process)  externally  developed  capabilities 
for  inclusion  into  the  baseline  tool. 

X 

7.  Publishes  a  coding  standards  and  style 
guide  with  which  all  externally  developed 
capabilities  are  required  to  comply. 

X 

X 

8.  Has  developed  and  implemented  a  quality 
assurance  process  that  rigorously 
evaluates  each  new  baseline  tool 
implementation  before  final  product 
release. 

X 

9.  Receives  and  expends  the  funds 
necessary  to  conduct  verification  and 
validation  tests  on  all  new  enhancements, 
and  thorough  regression  tests  on  all  new 
baseline  releases  to  ensure  past 
functionality  has  not  been  compromised. 

X 

- dPL 


M&S  Tool  Management  Success  Attributes  (3  of  3) 


M&S  Tool  Management  Success  Attributes: 

"The  M&S  Tool  Manager ..." 

Meeting 

Foreseeable 

Needs 

Integrating 

User-Developed 

Enhancements 

Model 

Accuracy 

(V&V) 

Customer 

Support 

1 0.  Updates  the  User's  Guide  and  /  or  Technical 
Reference  Manual  with  each  baseline 
enhancement  release,  including  constraints 
and  limitations. 

X 

X 

1 1 .  Receives  consistent  and  adequate  funding 
to  conduct  tool  baseline  maintenance, 
exclusive  of  baseline  enhancements,  to 
ensure  the  tool  remains  compatible  with 
current  software  and  hardware  products  used 
within  the  M&S  community. 

X 

12.  Provides  timely  customer  support  upon 
receiving  a  request  for  assistance  (i.e.,  a 
competent  and  adequately  staffed  Help 

Desk). 

X 
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Future  Work 


■  Develop  preliminary  set  of  recommended  actions  DoD  should  take 
to  improve  its  management  of  broadly-needed  M&S  tools 

■  Share  M&S  tool  management  success  attributes  and  preliminary 
set  of  recommended  actions  with  tool  managers  participating  in 
the  survey,  and  other  selected  members  of  the  DoD  M&S 
community 

■  Update  recommendations  based  on  comments 

■  Develop  list  of  desirable  characteristics  of  candidate  tools  to  be 
used  in  pilot  applications 

■  Produce  final  report  (now  targeted  for  February  2010) 


How  You  Can  Still  Participate 


■  If  you  are  a  government  or  industry  manager  of  a  broadly  used 
M&S  tool,  please  complete  the  survey  at 

http://outersurvevor.outer.ihuapl.edU/ss/wsb.dll/s/6qd 

■  Survey  should  take  10-15  minutes  to  complete 

■  If  you  have  prior  experience  in  managing  or  using  M&S  tools  and 
have  insights  on  best  practices  in  M&S  tool  management,  please 
complete  the  M&S  tool  user  survey  at 

http://outersurvevor.outer.ihuapl.edU/ss/wsb.dll/s/6qe 

■  Survey  is  similar  to  manager  survey,  but  not  tool-specific 


NDIA-  12th  Annual  SE 
Conference 

Live,  Virtual,  Constructive 
Architecture  Roadmap:  The  Quest 
for  Interoperability,  Standards,  and 

Reuse 

Gary  W  Allen,  PhD 
Project  Manager 
JTIEC 
& 

Amy  Henninger,  PhD 
Associate  Director 
M&SCO 
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Presentation  Agenda 


•  Introduction 

•  Where  are  we? 

•  How  did  we  get  here? 

•  Where  are  we  going? 

•  How  are  we  getting 
there? 

•  Why  are  we  doing  this? 

•  Conclusion 
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Where  are  we? 


•LVCAR  Study  Completed  SEP  2008 
•Provided: 

•Recommendations 
•Rationale 
•Business  Models 


•Phase 

•LVCAR  Implementation 
•Two  year  effort 
•Take  advantage  quick  starts 
•Involve  the  entire  M&S  community 
•Provide  exit  strategy 


How  did  we  get  here? 


Growing  Demand  For 
LVC  Interoperability 
Technical  &  Joint  Operations 


Broad  Proliferation  of 
Tools,  Standards,  Gateways 
Repositories,  etc 


Numerous,  Parallel 
Architectures 
(HLA,  DIS,  CTIA,  TENA) 
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How  did  we  get  here? 


•  Live,  Virtual,  Constructive  Architecture  Roadmap  (LVCAR)  Study 

•  Purpose:  “Develop  a  future  vision  and  supporting  strategy  for 
achieving  significant  interoperability  improvements  in  LVC 
simulation  environments.” 

•  Focus:  Three  dimensions  of  simulation  interoperability 

-  Technical  Architecture 

-  Business  Models 

-  Standards  Evolution  and  Management  Proce 

•  Precepts  That  Guide  Implementation 

-  Do  no  harm 

-  Interoperability  is  not  free 

-  Start  with  small  steps 

-  Provide  central  management 

•  Result:  A  set  of  recommendations  that  guide  HLT  S-C-2 
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How  did  we  get  here? 


Maintain  thte  Status  Quo 
k(“D(^Nothin||”)  (Strategy  1) 


Current 

Status 


Develop  A  New  Archl^ct^ 
(Strategy  5) 


Twitch  to  a 
>in\e,  newly- 
deweloped 
arofiitecture 


Enhance 
Interoperability  of 
Mixed-Architecture  | 
Events 
(Strategy  2) 


Encourage  and  Facilitate 
Architecture  Convergence 
(Strategy  3) 

Creation  of 


Select  One  oflthe 
Existing  Architectures 
(Strategy  4) 


/  a  logical  \ 

/  Single  X 

I  architecture 

/  architecture  \ 

\  (architecture  / 

(  implementation  ] 

\  of  architectures)  f 

\  emerges  from  J 

AlVswitch  to 
a  designated 
architecture 


existing  set 


Strategy  #2  will 
constitute  much  of  the 
initial  phase  in  executing 
option  #3 


(Ultimate  goal  is  a  single 
architecture  -  could  also 
be  a  smaller  set  of 
interoperable 
architecture 
implementations) 


Strategy  #2  is  a  viable 
branch  (exit  path)  from 
Strategy  #3,  if  warranted 
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Where  are  we  going? 


Focus:  Pursue  recommendations  identified  in  the  Live,  Virtual,  Constructive 
Architecture  Roadmap  (LVCAR)  report  sponsored  by  the  OSD  Modeling  & 
Simulation  Steering  Committee  (M&SSC) 


ACTIVITY 

Arch-Independent  Object 
Common  Capabilities  (Reuse) 
Architecture  Convergence 
Bridges  &  Gateways 
Management  &  Standards 
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Where  are  we  going? 


Approach 

-  Reduce  LVC  architecture  divergence  and  tool  proliferation 


-  Identify  organizational  and  structural  (e.g.  use  of  standards) 
options  to  better  manage  LVC  architecture  interoperability 


-  Create  reference  models  to  focus  data  and  service  reuse  efforts 


-  Explore  emerging  technology  issues  related  to  future  LVC 
architecture  performance  and  requirements 
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Where  are  we  going? 


*  Desired  Results 

-  Standardized  bridges  and  gateways  to  link  architectures 

-  Convergence  of  LVC  architecture  activities  and  reuse 
libraries 


Commonality  in  federation  templates,  object  models, 
engineering  processes 


Initiatives  to  pursue  translational  archit 
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o6  (f)  C/)  O 


How  are  we  getting  there? 


M 


LVCAR 

Study 


Identifies 

Areas 

of 

Investment 


(FY09  Phase  1) 


MANAGING 
THE  LVC 
ENVIRONMENT 


CONVERGE 
THE  LCV 
ARCH 


COMMON 

LVC 

CAPABILITIES 


COMMON  LVC 
GATEWAYS  & 
BRIDGES 


ARCH  IND 
OBJECT 
MODEL 


(FY10  Phase  2) 


Tool  Box 

Arch  Ind  Data  Formats 
Library/Depository 
Dev  Tools 
LVC  Capabilities 
STD  Gateways 
STD  Bridges 


Net-Centric 

•SOA 

•Sim  Interface 


(FY11  Phase  3) 


LVCAR 

Exit 

Strategy 


►Business 

Models 

►Policies 

►Standards 

►Utilities 


(FY12  ) 
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Why  are  we  doing  this? 
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Why  are  we  doing  this? 


12 


Summary 


•  JTIEC  and  M&S  CO  have  established  a  productive  and 
collaborative  working  relationship 

•  Currently  S-C-2  HLT  has  no  major  performance, 
schedule  or  cost  issues 

•  S-C-2  Project  Manager  focus  is  to  carry  forward  the 
recommendations  from  the  defining  LVC  Architecture 
Roadmap  study  to  the  ultimate  benefit  of  the  readiness 
of  our  warfighters. 
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Discussion 


HQU  did  the 

L VC  I  STANDARDS 
MEETING  GO? 


£ 

s 


Vb 

E 


I 


£ 


E 

3 

d 

£ 


D 


DID  YOU  CONVINCE 
83  COMPANIES  TO 
ADOPT  STANDARDS 
THAT  BENEFIT  ONLY  US 
WHILE  DOOMING  THE 


i 

rf 

LL 

£ 

£ 

r. 

i 

i 

n 

— 


S 

■ry 

ft 


OR  ARE  YOU 
A  CQWLETE 
FAILURE? 


CAN  I 
HEAR 
THOSE 
CHOICES 
AGAIN? 


Gary  w  Allen,  PhD 
S-C-2  HLT  Project  Manager 

Joint  Training  Integration  and  Evaluation  Center  (JTIEC) 

qarv.allen@us.armv.mil 

(O)  407-208-5607 
(C)  407-601-9838 


14 


Systems  Engineering 
Approach  to  Workforce 
Development 


29  Oct  09 


Jim  Miller 

Director  of  Engineering 
327  ASW/EN 
Phone:  (405)  736-4101 
james.c.miller@tinker.af.mil 


Systems  Engineering  Definition 


•  Systems  engineering  can  be  thought  of  as  the  problem- 
independent  principles  and  methods  related  to  the 
successful  engineering  of  systems. 

•  DOD  Definition:  SE  is  an  interdisciplinary  approach 
encompassing  the  entire  technical  effort  to  evolve  and 
verify  an  integrated  and  total  life  cycle  balanced  set  of 
system,  people,  and  process  solutions  that  satisfy 
customer  needs. 

•  INCOSE  Definition:  Systems  Engineering  is  an 
interdisciplinary  approach  and  means  to  enable  the 
realization  of  successful  systems... Systems  Engineering 
integrates  all  the  disciplines  and  specialty  groups  into  a 
team  effort  forming  a  structured  development  process 
that  proceeds  from  concept  to  production  to  operation. 


So  What? 


•  The  workforce  is  certainly  “problem 
independent”,  a  “method”,  a  “means”  and 
“related  to  successful  engineering” 

•  By  any  account,  an  organization’s 
engineering  workforce  is  one  of  the  keys  to 
successful  systems  engineering. 


Rather  than  the  usual  ad  hoc,  target-of- 
opportunity  approach,  an  organization  can 
apply  a  disciplined,  methodical  systems 
engineering  approach  to  successfully  develop 
the  engineering  workforce. 


327th  Aircraft  Sustainment  Wing 
-Responsibilities-- 


r  1402 
Aircraft  Mgd 
(289  Inactive) 


Air  Traffic 
Control  & 
Landing  Sys 

MgdJj# 


28,598 

Engines  Mgd 
50  types 


63  Weapon 
Systems 
33  ATCALS 


Commands 


fit  far  tasw^  !  fm 


^  203  USAF 
Bases 
42  FMS 


FYO^  JBfillffu 

185  Program 


FY09  ^ 

$15.7B 

$39B  Contracts 

Obligation  Managed 

.Authority  w  In  FY09 


Depot 

N,  T.  Maintenance 
Completed 


Nations 


Basic  Systems  Engineering  Process 


INPUTS 


Requirements 

Analysis 


Requirements 

Loop 


Verification 

Loop 


Functional 

Analysis/ 

Allocation 


Design 

Loop 


Analysis  & 
Control 


Design 

Synthesis 


OUTPUTS 


Input 


•  334  Engineers  in  the  327  ASW 

•  Scattered  across: 

•  6  different  organizational  groups 

•  19  different  squadron/supervisors 

•  30  different  weapon  systems 

•  Composed  of: 

•  6  different  engineering  disciplines 

•  46  various  years  of  experience 

•  0  standardized,  comprehensive  development 
plans 


Requirements 


•  Develop  the  327  Aircraft  Sustainment 
Wing's  Engineering  Workforce 

-  All  Inclusive . all  engineers 

-  Standardized . consistent  throughout  org.’s 

-  Comprehensive.. ..covers  all  tenets  of  development 

-  Individualized . allows  for  individual  needs 

-  Repeatable . new  employees,  each  year 

-  Measureable . for  mngt  &  improvement 


Basic  Systems  Engineering  Process 


INPUTS 


Requirements 

Analysis 


Requirements 

Loop 


Verification 

Loop 


Functional 

Analysis/ 

Allocation 


Design 

Loop 


Analysis  & 
Control 


Design 

Synthesis 


OUTPUTS 
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Functional  Analysis 


•  Needed  to  breakdown  what  is  meant  by 
“Workforce  Development” 

•  Used  the  Requirements  Loop  process 

•  Determined  the  components  are: 

•  Education 

•  Professional  Military  Education  (PME) 

•  Acquisition  Professional  Development  Program 
(APDP) 

•  Career  Broadening 

•  Promotions 

•  Awards 

•  Training 


Basic  Systems  Engineering  Process 


INPUTS 


Requirements 

Analysis 


Requirements 

Loop 


Verification 

Loop 


Functional 

Analysis/ 

Allocation 


Design 

Loop 


Analysis  & 
Control 


Design 

Synthesis 


OUTPUTS 
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Design  Loop 


•  Recognized  some  systems  exist 

•  Did  not  cover  all  7  components 

•  Often  not  current 

•  Difficult  to  use 

•  Needed  simple  means,  to  look  at  all 
components  and  whole  org  together 

•  Spreadsheet  (62  x  354) 

•  Cumbersome,  but  will  improve  later... 

•  Horizontal  for  individual 

•  Vertical  for  organization 
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Design:  Personal  Data 


Name _ Series _ AFSC _ Group _ Squadron 


•  Allows  sorting  by  name  and  org 

•  Allows  usage  by  supervisors 

•  Because  all  data  is  in  one  spot,  very  easy 
for  employee  and  supervisor  to  verify  data 
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Design:  Education  Data 


Have  Masters? 

Degree  type 

In  Work 

No 

•  Everyone  is  in  one  of  three  categories:  yes, 
no,  in-work  (because  can  take  almost  2  years 
to  complete  and  DP  systems  do  not  show 
“in-work”) 

•  Degree  Type  filled  only  if  “yes”  or  “in  work” 

•  Post-degree  work  does  not  indicate  currency 
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Design:  PME 


PME 

BDE 

IDE 

BDE 

In 

(BOB) 

(A  OB  O) 

(/\WG) 

Work: 

None 

Everyone  is  in  one  of  three  categories: 
yes,  no,  in-work  (because  can  take  almost 
2  years  to  complete  and  DP  systems  do 
not  show  “in-work”) 


•  Recently  big  push  to  have 

•  Employee  should  pursue  “grade 
appropriate”  PME  level 
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Design:  APDP 


Acq  Psn 

APDP 

Career  Level 

APDP  Cert 

Level  1 

Level  II 

Level  III 

Rqd 

Current 

Several  key  issues: 

-  Do  they  have  an  APDP  Certification 

-  Are  they  in  an  Acq  Coded  job  and  what  level? 

-  Is  employee  current  ? 

-  Are  they  ready  for  “next  level”? 

Interesting  note:  found  huge 
organizational  gaps  when  compared 

-  Ex:  org  A  at  95%  acq  coded,  while 

org  B  is  34% 


Design:  Awards 


Awards  FY09  FY10  FY11 


“Awards  and  Recognition”  always  cited  in 
surveys  as  a  top  3  problem  area 

Unfortunately,  tough  to  keep  up  with 

-  Information  has  to  be  updated  by  awards 
monitors  manually 

-  Labor  intensive  effort 

-  Looking  for  org  trend 


Design:  Career  Broadening 


Career 

Date  Arrived 

Broadening 

in  Current 

Employee 

Done? 

When? 

Job? 

F*  ro  m  ot  ed  ? 

•  Chief  Engineer  decides  if  Career 
Broadening  vs  just  a  move 

•  Date  Arrived  in  Current  Job  is  color 
indexed  (cell  is  filled) 

-  Green  if  <3  years 

-  Yellow  if  >3  years  but  <5  years 

-  Red  if  >5  years 

•  Promotions  tracked  separately 
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Design:  Continuous  Learning 


Year  1  Courses  |  Year  2  Courses  |  Year  3  Courses  |  Year  4  Courses 


SYS  182 

SYS  155 

SYS  028 

SYS  165 

SYS  172 

SYS  116 

CLE003 

CLE009 

SYS  161 

SYS  138 

SYS  185 

CLC041 

FPM101 

CLE011 

•  1 6  courses 

•  4  per  year 

•  All  CBT  so  no  travel  expenses 

-  Minimize  time  away  from  job 

•  Once  16  completed,  individualized 
training/specialization  starts 
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Basic  Systems  Engineering  Process 


INPUTS 


Requirements 

Analysis 


Requirements 

Loop 


Verification 

Loop 


Functional 

Analysis/ 

Allocation 


Design 

Loop 


Analysis  & 
Control 


Design 

Synthesis 


OUTPUTS 
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Analysis  and  Control 


•  Several  methods  used  for  analysis  and 
control: 

-  Annual  meeting  to  standardize/adjust  entire 
program 

-  Metrics  for  each  all  7  components 

-  Metrics  for  years  to  track  trends 

-  Metrics  to  compare  organizations 

-  Tool  to  be  used  by  supervisor  twice  a  year 
with  employee 

-  Metrics  displayed  to  upper  management  at 
least  quarterly 
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Control  Metric:  327ASW  Training 


327  ASW  Goal  80% 
Stretch  Goal  90% 


■  Classes  Taken  ■  Classes  Not  Taken 
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Control  Metric:  Acq  Certifications 


Basic  Systems  Engineering  Process 


INPUTS 


Requirements 

Analysis 


Requirements 

Loop 


Verification 

Loop 


Functional 

Analysis/ 

Allocation 


Design 

Loop 


Analysis  & 
Control 


Design 

Synthesis 


OUTPUTS 
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Outputs 


•  A  trained,  developed  Workforce 

•  Workforce  Development  Plan  provides: 

-  I  individualized  attention 

-  Standard  baseline 

-  Comprehensive  look 

-  Repeatable  process 

-  Measureable  data 

-  Monitored  by  upper  management 

•  Example:  ASW  achieved  96%  training 
goals  for  FY09 


Summary 


•  327  ASW  developed  tangible  systems 
engineering  process/plan  to  develop  the 
engineering  workforce 

•  Clear-cut,  tangible  process 

-  Will  apply  to  1300  ALC  engineers  in  FY10 

-  Plans  to  use  for  other  disciplines  (PM,  loggies,  etc...) 

•  Metrics  to  measure  progress  for  management 


I  n  Place  and  I  n  Use  Now 
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wi 


Questions? 
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Basic  Systems  Engineering  Process 


INPUTS 


Requirements 

Analysis 


Requirements 

Loop 


Verification 

Loop 


Functional 

Analysis/ 

Allocation 


Design 

Loop 


Analysis  & 
Control 


Design 

Synthesis 


OUTPUTS 
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Essential  Characteristics  of 
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Maier’s  Characterization  of  Systems  of  Systems 


Autonomous  constituents  with  independent  operations  and  management 

•  Includes  people,  organizations,  software  agents,  etc. 

•  Source  of  independent  actions  and  decisions 

Evolution... 

•  Independent  evolution  of  each  constituent  to  respond  to  new  technology  and 
mission  needs  at  its  own  pace  and  direction 

•  Evolution  of  the  whole  in  response  to  changing  demand 

Emergent  behavior 

•  “Whole  is  different  than  the  sum  of  the  parts” 

•  Indirect  and  cumulative  effects  of  influences,  actions,  interactions 

Maier,  Mark  W.  “Architecting  Principles  for  Systems  of  Systems,”  Systems  Engineering  1,  4  (1998):  267- 
284. 


— Sw  Assurance  in  a  SoS  World: 

Software  Engineering  Institute  Carnegie Mellon  Interoperability  Challenges 

—  ZM  ZM  o  Sledge,  October  2009 

©2009  Carnegie  Mellon  University 


Types  of  SoS* 


Recognized 
objectives, 
designated 
manager  and 
resources 


interact  more  or 
less  voluntarily  to 
fulfill  agreed 
central  purposes 


•  Integrated  SoS, 
built  and  managed 
to  fulfill  specific 
purposes 

•  Centrally  managed  to 
maintain  and  evolve 

•  Constituents 
independent  but 
subordinated  to 
centrally  managed 
purpose 


•  Constituents 
maintain  independent 
ownership,  objectives, 
funding,  etc 


•  Lack  central 
management 
authority  and 
centrally  agreed 
purpose 

•  Rely  on  relatively 
invisible 
mechanisms  to 
maintain  it 


•  Changes  based  on 
collaboration 
between  the  SoS 
and  the  constituent 


*  DoD  System  Engineering  Guide  for  System  of  Systems  Engineering  (Version  1 .0,  August  2008)  &  Maier 


— Sw  Assurance  in  a  SoS  World: 

Software  Engineering  Institute  Carnegie Mellon  Interoperability  Challenges 

—  ZM  ZM  o  Sledge,  October  2009 
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SoSSA  Assurance  Focus 


System  Assurance 

•  The  justified  confidence  that  a  system  functions  as  intended  and  is  free  of 
exploitable  vulnerabilities,  either  intentionally  or  unintentionally  designed  or 
inserted  as  part  of  the  system  at  any  time  during  the  life  cycle* 

Software  Assurance 

•  Software’s  contribution  to  system  and  SoS  assurance 

-  Software  assurance  in  the  context  of  a  system’s  mission  and  use 


*  Engineering  for  System  Assurance,  NDIA  System  Assurance  Committee,  2008, 

www.acq.osd.mil/sse/pg/guidance.html 
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Initiative  Scope  and  Goal 


Scope 

•  Large-scale  multi-user  adaptive  information  management  and  C2  systems  of 
systems  (SoSs) 

Goal:  Methods  and  practices  to  provide 

•  Justified  confidence  that  systems  of  systems  will  function  as  intended  in  their 
actual  environment  of  use  despite 

-  The  inevitable  presence  of  various  undiscovered  defects  and 
vulnerabilities 

-  Unanticipated  usage,  environmental  conditions,  reconfiguration,  or 
evolution 

•  Speedier  delivery  of  fielded  SoS  capability 
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Integration  &  Interoperability 


Software  Engineering  Institute 


Sw  Assurance  in  a  SoS  World: 

Carnegie  Mellon  Interoperability  Challenges 

^  Sledge,  October  2009 
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One  Aspect:  Integration  and  Interoperability 


Currently,  primarily  interoperability  issues  surfaced  at  integration  of  the 
SoS  for  test  and  evaluation  prior  to  fielding 

•  far  too  late  in  the  systems  engineering  lifecycle  to  effectively  and 
efficiently  deal  with  the  issues 


Additional  challenges  with  SoS 

•  underlying  constituent  systems  in  an  SoS  are  constantly  and 
independently  evolving 

•  producing  a  constant  state  of  evolutionary  and  continual  deployment 


Need  to  surface  (and  mitigate)  interoperability  and  integration  issues 
earlier  in  the  SoS  lifecycle 


Software  Engineering  Institute  Carnegie  Mellon 
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Premise 


Leverage  insight  from  prior  and  existing  DoD  SoS 

•  DoD  and  industry  sources 


Re 

•  interoperability  “failures”  (and  how  to  surface  interoperability  and  integration 
issues  earlier  in  the  SoS  lifecycle) 

•  what  practices  have  facilitated  better  and  quicker  integration 

•  were  there  software  approaches  that  could  have  helped  mitigate  the  issues 

•  were  there  associated  DoD  policy,  acquisition,  and  procedure 
challenges/barriers/incentives 

Assumed  anonymity/“genericized”  unless  explicit  permission  given 
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Overall  Findings 


Reluctance  to  discuss  SoS  interoperability  “failures’Ychallenges,  even 
with  anonymity 

Lack  of  “higher  level”  sharing  of  knowledge 

•  Software  engineering  issues,  risks  and  lessons  learned 

•  Organizational,  management  and  governance 

•  Analysis,  capture  and  dissemination 

•  Experience  (over  years) 

•  What  has  worked  and  what  has  not  (post  mortem) 

•  Time,  cost  and  “not  in  the  mainstream” 

Magnification  by  SoS  of  existing,  known  software  system 
problems  plus  new  and  emergent  problems 


Software  Engineering  Institute  Carnegie  Mellon 
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Some  Specific  Comments  from  Interviews 


Interoperability  claimed  but ... 

Find  problems,  do  workarounds  but  then  forget  about  problems  -  to  be 
discovered  again 

No  good  processes  that  look  at  interoperability  issues  (id,  avoid  or 
mitigate  them,  disseminate  solution  (collection  agency  or  repository)) 

Interoperability  “personality”  driven 

•  Individual  takes  it  on  to  identify,  document  and  work  with  programs  to  get  it 
resolved 

Different  standards,  interfaces,  etc. 

•  Surface  interoperability  issues  much  earlier  and  develop  mitigations  or 
solutions  (especially  cross  service) 

•  Find  the  right  people,  at  the  right  time,  at  the  right  level 

•  Even  within  service,  may  have  different  types  of  equipment  that  can’t  talk  to 
one  another 

•  Trying  to  avoid  dependence  on  one  company  (fair  share) 


— Sw  Assurance  in  a  SoS  World: 

Software  Engineering  Institute  Carnegie Mellon  Interoperability  Challenges 

—  ZM  ZM  o  Sledge,  October  2009 
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Specific  Comments:  Leveraging  the  Learning 
Curve 

Positive  experience  -  in  sustainment,  doing  things  early,  being  proactive 

After  action  reports,  other  lessons  learned,  “knowledge  base” 

•  Sometimes  the  knowledge  base  is  a  person  (personality  and  social  networks) 

•  “Human  interoperability” 

•  Attempting  to  institutionalize  it 

Earlier  in  the  life  cycle  -  going  against  grain 

•  Still  dealing  with  hardware,  beginnings  of  software  engineering,  do  some 
preliminary  software  interoperability 

•  Not  in  contract,  far  down  in  WBS 

•  Knowledgeable  people  “on  board”  earlier  -  avoid  mistakes  or  consider  what 
has  happened  in  similar  situations 
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Artifacts 


Currency,  existence,  completeness,  and  accessibility 

•  Architecture 

•  Design 

•  Rationale 

•  Assumptions 

•  Implicit  assumptions 

•  Not  machine-checkable 

•  Data  and  information 

•  Semantic/lexicons 

•  Access  to  and  incompatibility  of  information 

•  Different  tools 

•  Level  of  detail 

•  Critical  information  not  captured  in  artifacts 

•  What  is  critical,  what  becomes  critical  (based  on  changes) 
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Identified  Issues  for  Architecture/Architects 


Do  not  have  adequate  software  architecture  documentation  in  place 

•  Modification  to  what  the  system  is  interfacing  to 

•  Time  and  money  to  bring  “as  is”  architecture  documentation  up  to  date  and 
still  do  the  “to  be”  architecture  documentation 

Architect  needs  to  talk  directly  with  customer(s)  to  understand  expected 
use 

•  Uncover  interoperability  issues 

Similarly  architect  requires  timely  access  to  internal  corporate  subject 
matter  experts 

•  Share  expertise 
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Identified  Testing  Issues 


Mission  threads  do  not  reflect  current  operational  environment  reality 
Poor  systems  level  testing  done 
Changes  to  various  systems 

•  How  do  those  changes  affect  the  threads  and  tests 

Core  systems  -  one  simple  change  of  interface  standard  by  a  core 
system,  caused  many  problems  in  other  systems 

Challenge:  processes,  artifacts,  and  collaborations  in  systems  of 
systems  are  dynamic  and  ongoing,  not  static. 

•  Implies  continual  integration  and  test  are  necessary 

•  Interim  and  incremental  demonstration  of  interoperability,  SoS 
functionality,  and  SoS  capability 

Evaluation  and  leveraging  of  evidence  become  increasing  important 
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Identified  Practice  Issues 

Integration,  interoperability  -  mostly  considered  late  in  life  cycle 

•  Earlier  integration 

•  Allow  systems  to  come  to  test  floor/op.  environment  prior  to  formal  integration 

•  Interoperability  risk  reduction  exercises 

•  C4ISR  On-The-Move  (integrated  technology  demonstration) 

•  Tactical  Network  Topology  (field  experiment  exercise  environment) 

Specific  guidance  (usually  lower  level) 

•  Net-Centric  Enterprise  Solutions  for  Interoperability  [NESI] 

•  Cross  service  effort  (Army,  Navy,  DISA);  http://nesipublic.spawar.navy.mil 

•  “Body  of  architectural  and  engineering  knowledge  that  guides 

•  Design,  implementation,  maintenance  evolution  and  use  of  IT  portion  of  net- 
centric  solutions  for  defense  applications” 

•  E.g.  information  interoperability:  “To  be  able  to  share  information,  applications 
must  be  able  to  share  data  and  to  agree  on  its  meaning”  (access  to  data, 
semantic  match) 


— Sw  Assurance  in  a  SoS  World: 

Software  Engineering  Institute  Carnegie Mellon  Interoperability  Challenges 

—  ZM  ZM  o  Sledge,  October  2009 
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DoD  Policy,  Acquisition,  and  Procedure 
Challenges/Barriers/Incentives 

Most  SoS  are  not  Programs  of  Record 

•  Usually  no  specific  SoS  funding,  authority,  management  or  engineering 

•  At  best,  influence  the  new,  or  changes,  upgrades 

Individual  systems  do  not  consider  larger  context  (interfaces, 
interdependencies,  etc.) 

Constant  SoS  evolution,  continual  deployment 

•  Coordination,  collaboration  amid  change  and  turnover 

•  (Re)certification 

Incentives  and  rewards  focus  on  system,  not  SoS 

•  What  is  best  or  better  for  SoS,  may  not  be  optimal  or  desired  for  an  individual 
system 

•  Challenges  to  meet  system  milestones/deliverables 

•  (Early)Dissemination  of  (potential)  changes/problems  to  others  detrimental  to 
program/contractor 
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Request  to  Audience  (from  a  SoS  Point  of  View) 


Pointers  to  and  access  to  DoD  and  industry  sources  to  leverage  insight  re 

•  Interoperability  “failures”  (and  how  to  surface  interoperability  and  integration 
issues  earlier  in  the  SoS  lifecycle) 

•  What  practices  have  facilitated  better  and  quicker  integration 

•  Were  there  software  approaches  that  could  have  helped  mitigate  the  issues 

•  Were  there  associated  DoD  policy,  acquisition,  and  procedure 
challenges/barriers/incentives 

Additionally  seeking  insight  and  information 

•  How  conclusions  about  (software)  system  interoperability  could  be  developed 
faster  &  more  accurately  by  taking  advantage  of  evidence  gathered  throughout 
the  lifecycle 

•  Determine  what  evidence  could  be  provided  at  different  stages  and  how  it 
could  be  used  to  develop  justified  predictions  that  a  fielded  system  will  not 
experience  certain  types  of  interoperability  problems 
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Contact  Information 


Presenter: 

Carol  A.  Sledge,  Ph.D. 

Research,  Technology,  and  System 
Solutions 

Telephone:  +1  412-268-7708 
Email:  cas@sei.cmu.edu 

World  Wide  Web: 

www.sei.cmu.edu 


Visit  the  SEI  display  at  the 
Regatta  Pavilion 


Software  Engineering  Institute 


U.S.  mail: 

Software  Engineering  Institute 
Carnegie  Mellon  University 
4500  Fifth  Avenue 
Pittsburgh,  PA  1521 3-261 2 
USA 


Sw  Assurance  in  a  SoS  World: 
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Improving  Systems  Engineering  Curriculum  Using 
a  Competency-Based  Assessment  Approach 


Alice  Squires, 
alice.squires@stevens.edu 

School  of  Systems  and  Enterprises 

Stevens  Institute  of  Technology 
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Why  Competencies? 

To  meet  government/industry  needs  ...  today. 


Government  and  Industry  groups  have  defined 
knowledge,  skills  and  abilities  (competencies) 
that  are  important  to  their  success. 

-  behaviors,  attitudes,  attributes  =  also  competencies 

-  performance/output  minimum  =  competences 


Curriculum  can  be  designed  to  address  these 
competencies. 

-  learning  objectives 

-  course  content 


-  activities 


An  approach  to  teaching  and  learning  that  is 
based  on  the  successful  student  achieving  a 
specific  level  of  proficiency  in  a  specific  set  of 
competencies. 

Compare: 

-  current  level  -  ‘as  is’ 

-  desired  level  -  ‘to  be’ 

Identify  gaps  -  focus  areas 

Put  a  plan  in  place  to  bridge  the  gap 
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What  is  a  Competency-Based  Approach? 
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An  Example  of  ‘Individual’ 
SE  Competency  Models  —  FAA 


•  FAA  SE  Manual,  October  11,  2006: 

http://www.faa.gov/about/office  org/headauarters  offices/ato/s 

ervice  units/operations/svsengsaf/seman/ 

•  Armstrong,  J.  R.,  &  Henry,  D.  (2009).  Competencies  required  for 
successful  acquisition  of  the  next  generation  air  transportation  system.  In 
IEEE  syscon  2009 ,  3rd  annual  IEEE  international  systems  conference, 
Vancouver,  Canada,  march  23-29,  2009. 

•  Armstrong,  J.,  Henry,  D.,  &  Pyster,  A.  (2009,  September  8).  Systems 
engineering,  systems  integration,  and  software  engineering  competencies 
required  for  successful  acquisition  of  the  next  generation  air 
transportation  system.  School  of  Systems  and  Enterprises,  Stevens 
Institute  of  Technology,  Castle  Point  on  Hudson,  Hoboken,  NJ. 

•  Turner,  R.,  Verma,  D.,  &  Weitekamp,  W.  (2009,  August  31).  The  next 
generation  air  transportation  system  (nextgen).  School  of  Systems  and 
Enterprises,  Stevens  Institute  of  Technology,  Castle  Point  on  Hudson, 
Hoboken,  NJ. 

Other  examples:  MITRE,  SAIC,  BAE  Systems,  Nokia,  Boeing,  LM,  etc.... 
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Examples  of  ‘Jointly’ 
Developed  SE  Competency  Models 


DAU  Systems  Planning,  Research,  Development 
and  Engineering  (SPRDE)-SE/PSE  Model: 

https: //acc.dau.mil/CommunitvBrowser.aspx?id=31 5691 

-  Analytical  (13),  Technical  Management  (12),  Professional  (4) 

INCOSE  UK:  Systems  Engineering  Competencies 
Framework  and  Guide  to  Competency  Evaluation 

-  final  due  out  Oct  2009 

-  Holistic  LifeCycle,  Systems  Thinking,  Systems  Engineering  Mgmt 

NASA/ Industry:  See 

http://www.nasa.gov/pdf/303747main  Systems  Engineerina  Comp 

etencies.pdf 


Model  shown  and  used  in  this  presentation 


International  Academy  of  Astronautics  (IAA): 
Space  industry  SE  competency  model: 

-  10  Competency  Areas 

-  37  Capabilities  within  these  ten  areas 

-  4  Proficiency  Levels 

Successfully  Leveraged  to  Develop  Stevens’  SSE 
Domain  Centric  Space  Systems  Engineering 
Masters  Program 

Used  to  Assess  Legacy  Core  Courses  for  Stevens’ 
SSE  SE  Discipline  Centric  Masters  Program 
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NASA/Industry  SE  Competency  Model 


Ten  Competency  Areas 
(#  of  capabilities) 


1.  Concepts  and  Architecture  (4) 

2.  System  Design  (4) 

3.  Production,  Product  Transition  and  Operations  (6) 

4.  Technical  Management  (8) 

5.  Project  Management  and  Control  (4) 

6.  Organizational  Environments  (3) 

7.  Human  Capital  Management  (2) 

8.  Security,  Safety  and  Mission  Assurance  (2) 

9.  Professional  and  Leadership  Development  (3) 

10.  Knowledge  Management  (1) 
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•  Level  I  -  Participate  (Know):  Performs  fundamental  and  routine  SE 
activities  while  supporting  a  Level  ll-IV  systems  engineer  as  a 
member  of  a  project  team 

•  Level  II  -  Apply  (Perform):  Performs  SE  activities  for  a  subsystem  or 
simple  project  (e.g.  no  more  than  two  simple  internal/external 
interfaces,  simpler  contracting  processes,  smaller  team/budget, 
shorter  duration) 

•  Level  III  -  Manage  (Lead):  Performs  as  a  systems  engineer  for  a 
complex  project  (e.g.  several  distinct  subsystems  or  other  defined 
services,  capabilities,  or  products  and  their  associated  interfaces) 

•  Level  IV  -  Guide  (Strategize):  Oversees  SE  activities  for  a  program 
with  several  systems  and/or  establishes  SE  policies  at  top 
organizational  level. 


*from  NASA’s  Academy  of  Program/Project  and  Engineering  Leadership  (APPEL) 
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Space  Systems  Engineering 

Example 


Fundamentals  of 
Systems 
Engineering 

System  Architecture 
and  Design 

Designing  Space 
Missions  and 
Systems 

Mission  and  System 

Design  Verification 

&  Validation 

Systems  Integration 

Project  Management 

of  Complex 

Systems 

Human  Spaceflight 

Space  Launch  and 

Transportation 

Systems 

Cost  Effective 

Space  Mission 

Operations 

Crew  Exploration 

and  Vehicle  Design 

Modeling  and 

Simulation 

Design  for 

Reliability, 

Maintainability  and 

Supportability 

Decision  and  Risk 

Analysis 

Core 

Specialty 

1.0 

1.1 

X 

X 

X 

X 

X 

X 

1.2 

X 

X 

X 

X* 

X 

X 

X 

X 

1.3 

X 

X 

X 

X* 

X 

X 

X 

X 

X 

X 

X 

1.4 

X 

X 

X* 

X 

X 

2.0 

2.1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

2.2 

X 

X 

X 

X 

X 

X 

X 

X 

X 

2.3 

X 

2.4 

X 

X 

X 

X* 

X 

X 

X 

X 

X 

3.0 

3.1 

X 

3.2 

X 

X 

X 

3.3 

X 

X 

X 

X 

3.4 

X 

X 

X 

3.5 

3.6 

X 

X 

X 

x*  -  for  enabling  systems 
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The  Team:  Years  of  Experience 


Government/ 

Industry 

Research/ 

Academia 

SE 

Related 

TOTAL 

1 

42 

1 

43 

43 

2 

35 

5 

35 

40 

3 

27 

3 

23 

30 

4 

29 

3 

20 

32 

5 

27 

6 

30 

33 

6 

26 

12 

18 

38 

7 

21 

10 

10 

31 

8 

6 

6 

5 

12 

9 

30 

9 

39 

39 

10 

26 

7 

23 

33 

11 

23 

4 

13 

27 

12 

27 

4 

15 

31 

13 

9 

12 

20 

21 

328 

82 

294 

410 
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•  Select  the  Competency  Model  to  Use 

•  Validate  the  ‘Critical'  Competencies 

•  Identify  the  ‘As  is’  State  of  the  Curriculum 

•  Identify  the  ‘To  Be’  State  of  the  Curriculum 

•  Summarize/ Evaluate  the  Gap  Areas 

•  Put  a  Plan  of  Action  in  Place  to  Assess  those 
Gaps 

•  Revisit! 
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Systems  Engineering 

IAA  Global 

Stevens 

Capabilities 

’Apply' 

’Apply’ 

3.0  Production,  Product  Transition  and 
Operations 

3.1  Implement  the  Product 

Optional 

5 

3.2  Integrate  System 

Critical 

Critical 

10 

3.3  Verify  the  System 

Critical 

Critical 

9 

3.4  Validate  the  System 

Necessary 

Critical 

9 

3.5  Transition  the  System 

Optional 

4 

3.6  Conduct  Operations 

Necessary 

2 
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Determine  Current  State:  ‘As  is’ 


Stevens  Institute  of  Technology: 
Critical  Systems  Engineer 
Capabilities 

Fundamentals  of  Systems  and 

Software  Engineering 

System  Architecture  and 

Design 

Systems  Integration 

Project  Management  of 

Complex  Systems 

3.0  Production,  Product  Transition  and 

Operations 

3.2  Integrate  System 

Low 

Medium 

3.3  Verify  the  System 

Medium 

Medium 

3.4  Validate  the  System 

Medium 

Medium 

School  ftf 
SyhtcniRA  Einlcqiris-L-i 


STEVENS 


llnstitute  of  Technology! 


Determine  Desired  State:  ‘To  Be’ 


Stevens  Institute  of  Technology: 
Critical  Systems  Engineer 
Capabilities 

Fundamentals  of  Systems  and 

Software  Engineering 

System  Architecture  and 

Design 

Systems  Integration 

Project  Management  of 

Complex  Systems 

3.0  Production,  Product  Transition  and 

Operations 

3.2  Integrate  System 

Low 

High 

3.3  Verify  the  System 

Medium 

High 

3.4  Validate  the  System 

Medium 

High 

School  of 

SyhtcnifiA  Einlcqiris-L-s 


Evaluate  Gaps 


STEVENS  = 


Institute  of  Technology  j 


Stevens  Institute  of  Technology: 
Critical  Systems  Engineer 
Capabilities 

1  circle  =  Low 

3  circles  =  Medium 

5  circles  =  High 

•  =  current  ?as/is' 

O  =  ?to/bef 

Fundamentals  of  Systems  and 

Software  Engineering 

System  Architecture  and 

Design 

Systems  Integration 

Project  Management  of 

Complex  Systems 

3.0  Production,  Product  Transition  and 
Operations 

3.2  Integrate  System 

• 

•••oo 

3.3  Verify  the  System 

•  •• 

•••oo 

3.4  Validate  the  System 

•  •• 

•••oo 

School  of 
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Develop  an  Action  Plan 


Systems  Integration 

SE 

Capabilities 

Gaps 

Examples  of  Potential  Actions  To  Address  Gaps 

3.0  Production,  Product  Transition  and  Operations 

3.2  Integrate 
Systemp 

•••oo 

Increased  emphasis  will  be  placed  on  integration 
strategies  to  address  interface  risks  early  in  a 
program. 

3.3  Verify  the 
Systemp 

•••oo 

May  introduce  Design  of  Experiments  (DOE)  in  this 
context. 

3.4  Validate 
the  Systemp 

•••oo 

Increased  emphasis  on  tie  to  risk  of  non-acceptance  of 
system  by  the  customer. 

Related  Journal  Papers 
To  Be  Published 


Squires,  A.,  Larson,  W.,  and  Sauser,  B.  (2010). 
“Mapping  Space-Based  Systems  Engineering  Curriculum 
to  Government-Industry  Vetted  Competencies  for 
Improved  Organizational  Performance”,  Systems 
Engineering,  13(2  or  3),  TBD. 

Squires,  A.,  and  Larson,  W.  (2009).  “Improving  Systems 
Engineering  Curriculum  Using  a  Competency-Based 
Assessment  Approach”,  Special  Issue  on  Systems 
Engineering  Education  of  the  International  Journal  of 
Intelligent  Defence  Support  Systems  (IJIDSS), 

TBD(TBD),  TBD. 
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Generating  Visual  and 
Interactive  Output  from  System 

Engineering  Tools 

John  Schatz  /  SPEC  Innovations 
29  October  2009 


Overview 


•  Methodology 

•  KBAD  Schema 

•  Information  Capture  Process 

•  Risk  Matrix  Visualization 

•  TPM  Capture  and  Visualization 

•  Expandable-Collapsible  Tree  Visualization 

•  Geo-Spatial  Visualization 


INNOVATIONS 


Methodology 


Modify  SE  Knowledge- 
Base  Schema 


Extract  Data  from  SE  Export  Data  in 

Knowledge-base  Modified  Format  Determine  Outputs 

(RTF,  HTML,  XML) 


Capture  Information  in 
SE  Knowledge-Base 


Execute  Criteria 
Based  Business  Logic 


Output  Loop 


Execute  Output  Loop 


KBAD  Schema 


KBAD*  Element 

CORE  Elements 

Rationale 

Action 

Function/Operational  Activity 

Provide  overall  class  for  actions 

Artifact 

Document 

Recognized  not  just  documents 

Asset 

Component/Operational  Element 

Provide  overall  class  for  assets 

Characteristic 

type  of  Requirement 

Way  to  capture  metrics  and  other 
characteristics  of  an  element 

Cost 

attribute  of  Component 

Broadens  capture  of  costs 

Input/Output 

Item/Operational  Information 

Clearer  name 

Issue 

Issue 

Same 

Link 

Link/Needline 

Provide  overall  class  for  transmission 

Location 

none 

Captures  geolocation  information 

Risk 

Risk 

Same 

Statement 

type  of  Requirement 

Clearer  name 

Time 

attribute  of  Function 

Broadens  capture  of  times 

The  KBAD  Schema  supports  the  capture  of  data 
items  and  relationships  utilized  in  the  examples. 


*  Knowledge-Based  Analysis  and  Design 


4 


Capture  Information  in  SE 
Knowledge-Base 


%  Captlifand  Anai§§e  Related  Documents 


[fill  Requirements  Analysis 
□  Functional  Analysis 


I.- iVr'^ 


5.  Develop  the  Operational  Context  Diagram 

1 _ 1 

6.  Develop  Operational  Scenarios 

m 

7.  Derive  Functional  Behavior 

8.  Derive  System  Elements 

and  Control 


9.  Allocate  Functions  to  System  Elements 

4.  Capture  Constraints 

10.  Prepare  Interface  Diagrams 

3.  Identify  Existing/Planned  Systems 

14.  Provide  Options 

jlj2; j  peflciriii;  iAih#ys;i$  i 


11.  Define  Resources,  Error  Detection  &  Recovery 


13*  Develop  Operational  Demonstration  Master  Plan 


;  jl$;  p<jfjtliicit;|fiailiei-i6|ff  jAifijiiysjeS  i 


16.  Generate  Operational  and  System  Architecture  Graphics,  Briefings  and  Reports 


The  KBAD 
middle-out 
approach  has 
been  proven  on 
a  variety  of 
projects. 


Time 


Risk  Matrix  Example 


Risk  Characterization  Summary 

Cl  C2  C3  C4  C5 


Consequence  Impact:  Cl  =  Negligible.  C2  =  Minor.  C3  =  Moderate.  C4  =  Serious.  Ci  =  Critical 
Likelihood:  LI  =  (0-10L  L2  =  (10-40),  L3  =  (40450),  L4  =  (60-90),  Li  =  (90-100) 

Risk  Characterization:  Green  =  Low,  Yellow  =  Moderate,  Red  =  High 


Risk  Matrix  Example  -  Logic 


1.  Extract  Risks  of  interest. 

2.  Create  lists  of  risks  for  each  Risk  Matrix  cell  by  examining  the  risks' 
consequences  and  likelihoods. 

3.  Begin  streaming  RTF  file  up  to  first  cell. 

4.  Set  cell  color.  The  cell  colors  are  fixed. 

5.  Insert  risks  for  given  cell. 

6.  Etc. 


TPM  Example 


Technical  Performance  Measures 

Current 

Projected 

□ 

Threshold 

Objective 

□ 

Units 

Imp.Dir. 

Hardware  Assets 

SBPG  Ground  Element 

TPM:  Gr  oun  d  El  em  cut  MTTR 

[  30.0 

15.0  | 

□ 

minutes 

Negative 

SBPG  On-Orbit  Element 

TPM:  On-Orbit  Element  Lifespan 

13.0 

15.2 

n 

10.0 

15.0 

□ 

years 

Positive 

TPM:  On  -  Orbit  El  em  ent  Tran  smi  s  si  on  Effi  ci  ency 

0.65 

H 

0.65 

0.75 

H 

Positive 

TPM:  On-  Orbit  El  em  ent  Wei  ght 

5SO0.0 

h 

6000.0 

5000.0 

□ 

kg 

Negative 

Sy  ste  m  Functio  us 

Execute  Maneuver  Commands 

TPM:  Characteristic^  01 

13.2  I 

□ 

11.0 

13.0  | 

□ 

seconds 

Positive 

Issue  Maintenance  Alert 

TPM:  Characteristic^  02 

0.95 

0.97 

□ 

I  0.95 

0.98  | 

n 

1 

Positive 

Collect  Solar  Energy 

TPM:  Characteristic 003 

0.65 

0.66 

□ 

0.6 

0.65 

n 

i 

Positive 

Figure  1  SBPG  C  o  ntext  Pe  rfo  r  ma  nc  e  Pa  r  a  mete  r  s  Matrix 

Value  Characterization:  Green  =  Exceeds  Objective.  Yellow  =  Between  Threshold  and  Objective.  Red  =  D  oes  Not  Meet  Threshold 


TPM  Example  -  Logic 


1.  Extract  TPMs  for  systems  of  interest. 

2.  Begin  streaming  RTF  file  up  to  first  System  row. 

3.  Insert  System  name. 

4.  Stream  up  to  the  system's  first  TPM. 

5.  Insert  TPM  name. 

6.  Compare  current  and  projected  values  against  threshold  and  object 
values  taking  improvement  direction  into  account. 

7.  Determine  cell  color  based  on  predetermined  criteria. 

8.  Insert  color  coded  cells  with  current  and  projected  values. 

9.  Etc. 


VATIONS 


Expandable-Collapsible  Tree 

Example 


£  101  Component  A.1  [101]  Downstream  Impact  Tree  -  Internet  Explorer  provided  by  Dell 


_ )  -  !  0  C:\Users\schatzjo\Desktop\NNP\101  Component  A.1  [101]  •  Downstream  Impact  Tree.htm 


a  «  101  Component  A1  [101]  Downstream  Impact  Tr— 

went  A.1  [101]  Downstream  Impact  Tree 

[EXPAND  TREE]  [EXP^’TOl2]  [S^Os'.  ALL  DETAILS]  [HIDE  A—  DETALS] 

Smponent  A.1  [101]  [show  details] 


4$  101  Component  A.1  [101]  Downstream  Impact  Tree^k|emet  Explorer  provided  by  Dell 

^  C:\Users\schatzjo\Desktop\NNP\101  Component  A.1  [101]  -  Downstream  Impai 


ji  ^  101  Component  AJ.  [101]  Downstream  Impact  Tr... 


h 


101  ComponemjLJ-4-liUjDownstreai 

:  .:r 

[-]  101  Component  a.1  [iuij  [shcwdetails] 

111  Component  AZ.l  [111]  [show  details] 

121  Component  Z.l  [121]  [show  details] 

[-]  1191  Component  AZ.91  [1191]  [show  details] 

1291  Component  Z.91  [1291]  [show  details] 


101  Component  A.1  [101]  Downstream  Impact  Tree  -  Internet  Explorer  provided  by  Dell 

C:\Users\schat2jo\Desktop\NNP\101  Component  AJ  [101]  -  Downstream  Impact  Tree.htm 


£  <*  4$  101  Component  AJ  [101]  Downstream  Impact  Tr... 


101  Component  A.1  [101]  Downstream  Impact  Tree 

[EXPAND  TA.EE ]  [EXPAND  TO  L2]  [SHOW  ALL  DETAILS]  [HE*  ALL  DETAILS] 

[-]  101  Component  A.1  [101]  [hide  details] 

Organization:  Organization_OOl 
Location:  Location_00l,  Arlington,  VA,  USA 
IP  Address:  192.168.1.3 

111  Component  AZ.l  [111]  [hide details] 

Organization:  Organization_OOl 

Location:  Location_002,  Philadelphia,  PA,  USA 

IP  Address:  135.73.72.22 

121  Component  Z.l  [121]  [nioe details] 

Organization:  Organization_OOl 
Location:  Location_003,  Kaiserslautern.  FDR 
IP  Address:  102.104.2.3 

[-]  1191  Component  AZ.91  [1191]  [hide details] 

Organization:  Organization_001 
Location:  Location_002,  Philadelphia,  PA,  USA 
IP  Address:  133.2.200.23 

1291  Component  Z.91  [1291]  [hide  details] 

Organization:  Organization_003 
Location:  Location_003,  Kaiserslautern,  FDR 
IP  Address:  180.181.55.57 


Expandable-Collapsible  Tree 

Example  -  Logic 


1.  Write  JavaScript  and  CSS  files. 

2.  Extract  nodes  in  an  interconnected  nodal  network. 

3.  Generate  index  file  of  all  nodes. 

4.  Iterate  through  nodes  doing  the  following  for  each: 

1.  Generate  expandable  tree  branches  and  leaves.  Prevent  closed 
loops. 

2.  Begin  streaming  HTML  file  with  JavaScript  and  CSS  files  referenced. 

3.  Store  controls  and  tree  data  in  JavaScript  node  array. 

4.  Store  starting  positions  in  JavaScript  position  array. 

5.  Encode  nodes  into  HTML  file  as  absolutely  positioned  items  with 
embedded  JavaScript  commands  to  access  Document  Object  Model 
(DOM)  for  hiding  or  showing  nodes. 


Geo-Spatial  Example 


Lemon  Grove  ' 

Qv-rti  • 

Spring  Valley 

I ! 


National  City 


<  r.  Chula  Vista 


■ata  SIO,  MOM. -u  s,  Hhf  NGA.  GEBCO  a..';  •  4  I 

(§2009  Tele  Atlas 

■:  --  .  „  1  | 

s®  2009  Google  '  .  v  ■  - 1  z  .  _  , 

□  q  Imperial  Beach 


Google  Earth 


,  i  a  i^a»| ' 


'®jM.  a  iTi  rgija] 


q  S.atwe 


Fly  To  Find  Businesses  Directions 


Fly  to  e,g,r  Tokyo,  Japan 
San  Diego,  CA|  t 


v  Search 


E..;  San  Diego.  CA 


v  Places 


|  Zeodin  LO  Alpha  -  Build  1  -  Wed  Oct  07  14:44:31  EOT  2009  -  SPEC  Proprietary  Software 


ffl  Report  m  Database  w  Script 


i  Jew 


Number  Name 


Description 


Bements 

- 1 

All  Elements 

1 

Acronym 

Action 

Artifact 

Asset 

Characteristic 

Cost 

Input/Output 

Issue 

Link 

_ 1 

Location 

Multimedia 

1 

Risk 

Statement 

Time 

Schema 

J 

1.1  San  Diego,  CA  Location  of  Conference  Location 


»  Edit  Location  Element 


Properties  Attributes  Relationships  Configuration 


Number  1 1.1 

Name:  San  Diego,  CA 

Location  of  Conference 


Add  Content 


zeooTn 


Type 


Date  Created 


Date  Modified 


1.2 

Washington,  DC 

1  U  S  Capital 

Location 

Wed  Oct  07  14... 

Wed  Oct  07  14 

1  3 

New  York,  New 

U  S.  Financial  Center 

Location 

Wed  Oct  07  14 

Wed  Oct  07  14 

Description: 


""[  Save  ["]  Clone  ]*]  Delete  j  |  Reset  | 


Geo-Spatial  Example  -  Logic 


r  i 

XML  File 

L _ J 


is  imported  into 


Zeodin 


exports  to 


KML  File 


is  opened  with 


Google 

Earth 


1.  Generate  KML  header  information. 

2.  Extract  assets  from  SE  Knowledge- 
Base. 

3.  Iterate  through  assets  streaming 
asset  specific  KML. 


Summary 

•  Use  of  other  products  for  visualization  is 
necessary,  since  most  SE  tools  provide  poor 
graphics  for  a  general  audience 

•  Output  from  COTS  Products  can  be  modified  to 
enhance  visualization 

•  Most  tools  provide  scripting  that  enable  creative 
visualization 


An  Introduction  to  Influence 
Maps:  Foundations, 
Construction,  and  Use 


Jim  Smith 

NDIA  Systems  Engineering  Conference 
October  29,  2009 


Software  Engineering  Institute 


Carnegie  Mellon 


©2009  Carnegie  Mellon  University 


Overview 


This  presentation  will  provide  an  overview  of  Influence  Maps  (IMs),  a 
graph-based  technique — central  to  Influence  Mapping  Analysis  (IMA) — 
for  understanding  system-of-systems  interoperability  issues 

Topics  include: 

•  Foundations  for  IMs 

•  Construction  and  use  of  IMs  for  governance-  and  acquisition-related 
interoperability  risks 


Software  Engineering  Institute 


Carnegie  Mellon 


Jim  Smith:  Introduction  to  IMs 
NDIA  systems  Engineering  2009 

©2009  Carnegie  Mellon  University 


Multiple  Perspectives  on  System  of  Systems  -1 


SoS  are  a  collection  of  integrated  and  interoperable  hardware  and 
software  entities  providing  capabilities  that  fulfill  specific  functional  and 
operational  needs 


But... systems  of  systems  are  more  than 
interoperating  hardware  and  software  systems 


= -  Software  Engineering  Institute  Carnegie  Mellon 


Jim  Smith:  Introduction  to  IMs 
NDIA  systems  Engineering  2009 

©2009  Carnegie  Mellon  University 


Multiple  Perspectives  on  System  of  Systems  -2 


An  SoS  is  a  collection  of  people  and  organizational  entities  involved  in 
acquiring  and  composing  “systems  of  systems”  that  provide  capabilities 
to  fulfill  specified  functional  and  operational  needs 


People  systems  are  as  important  as  technical  systems 


Software  Engineering  Institute  Carnegie  Mellon 


Jim  Smith:  Introduction  to  IMs 
NDIA  systems  Engineering  2009 

©2009  Carnegie  Mellon  University 


Multiple  Perspectives  on  System  of  Systems  -3 


SoS  provide  capabilities  that 
enable  a  collection  of 
operational  users  to  achieve  the 
effects  they  need  to  meet  their 
business/mission  goals 

•  Evolves  to  enable  dynamically 
changing  operational  effects 
within  the  operational  user’s 
context  of  use 

•  Is  likely  to  use  technical  and 
organizational  assets  outside  of 
the  original  design  context 


Operational  Effects/ 
User  View 


Jim  Smith:  Introduction  to  IMs 


Software  Engineering  Institute  CarnedeMellon  NDIA  systems  Engineering  2009 

W  W  Pomania  MollAn  llniwarcitw 


2009  Carnegie  Mellon  University 


r 


v 


Key  Point:  Systems  of  Systems  Result  from 
Interrelationships 


The  composition  of  capabilities 
with  users  and  operational 
processes  that  achieves 
desired  operational  effects  for  a 
particular  context  of  use 


The  people,  organizations, 
and  interrelationships 
associated  with  building, 
acquiring,  fielding,  and 
evolving  systems  of  systems 


Aggregation  of  systems, 
hardware  or  software 
components,  and  other 
devices  to  provide 
operational  capability 


Software  Engineering  Institute  Carnegie  Mellon 


Jim  Smith:  Introduction  to  IMs 
NDIA  systems  Engineering  2009 

©2009  Carnegie  Mellon  University 


Understanding  Interrelationships  and  Influence 


Various  techniques  exist  to  represent  interrelationships  in  socio- 
technical  systems,  including: 

•  Network  diagrams  •  IDEF0/IDEF3 

•  Functional  Flow  Block  Diagrams  •  PERT  Charts 

•  Conceptual  Graphs 

Challenges  in  applying  to  systems  of  systems 

•  Complexity  of  resulting  representation 

•  Difficulties  in  representing/reasoning  about  “background”  knowledge 

•  Problems  in  representing/reconciling  conflicting/contradictory  influences 

Influence  Maps  (IMs) — and  IM  Analysis  (IMA) — provides  a  simple  way  to 
identify,  understand,  and  analyze  influence  interrelationships 

•  Permits  the  discovery  of  influences  that  impact  governance,  acquisition,  and 
engineering  for  systems  of  systems 

•  Supports  the  identification,  characterization,  and  reconciliation  of  ambiguous 
and  contradictory  influences 

•  Input  to  formal  analysis/reasoning  framework  and  decision  aids 


Software  Engineering  Institute  Carnegie  Mellon 


Jim  Smith:  Introduction  to  IMs 
NDIA  systems  Engineering  2009 

©2009  Carnegie  Mellon  University 


Key  Concepts  of  I  Ms  and  IMA 


Influence  mapping  analysis  is  built  around  several  key  concepts 

•  Use  of  IMs  to  identify,  characterize,  and  understand  influence  relationships 

•  Resolving  divergent  perceptions  of  the  actual  conditions:  the  so-called 
“ground  truth” 

•  Analyzing  patterns  of  influence  relationships  for  indicators  of  SoS  risks 

•  Use  of  a  contextually-driven  discovery  process 


Software  Engineering  Institute 


Carnegie  Mellon 


Jim  Smith:  Introduction  to  IMs 
NDIA  systems  Engineering  2009 

©2009  Carnegie  Mellon  University 


Understanding  the  Relationships  Implied  by  the 
SoS  Perspectives 


Relationships 

among 

stakeholders 


Relationships 

among 

goals/purpose 


Relationships 
among 
constituents 


Context(s)  of 
use 


Software  Engineering  Institute 


Carnegie  Mellon 


Jim  Smith:  Introduction  to  IMs 
NDIA  systems  Engineering  2009 

©2009  Carnegie  Mellon  University 


9 


Relationship  Characteristics  of  Systems  of 
Systems 


Stakeholder  volatility 
Stakeholder  diversity 

Stakeholder  autonomy 

Diversity  of  governance 
frameworks 

Centralization  of  control 

Flexibility/adaptability  of 
governance  frameworks 

Coherence  of  incentives 


Degree  of  emergence  of  capabilities 


Volatility  of  demand 
Variety  of  demand 


Context(s)  of 
use 


Constituent  diversity  Volatility  of  composition 

Constituent  volatility  Range  of  capability  provided 

Independent  evolution  of  constituents 


=  Software  Engineering  Institute  Carnegie  Mellon 


Jim  Smith:  Introduction  to  IMs 
NDIA  systems  Engineering  2009 

©2009  Carnegie  Mellon  University 
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Forms  of  Interrelationships 


The  interrelationships  in  socio-technical  systems — and 
their  influence — vary  widely: 

•  Contract  language 

•  Statutory/regulatory  requirements 

-  Defense  Appropriation  Act 

-  HIPAA 

-  Federal  Acquisition  Regulations 

•  Technical  requirements 

•  Reporting  requirements 

•  “Giver/receiver”  relationships 

•  Funding 

•  Individual  trust 


the  workings  of 


= -  Software  Engineering  Institute  Carnegie  Mellon 


Jim  Smith:  Introduction  to  IMs 
NDIA  systems  Engineering  2009 

©2009  Carnegie  Mellon  University 


Managing  Divergent  Perceptions  -1 


SoS  participants  can  have  a  different  understanding  of  relevant 
influence  relationships 

•  Important  influence  relationships  are  often  implicit,  or  only  tacitly 
acknowledged 

These  inconsistencies  can — and  frequently  do — lead  to  unfortunate 
technical  and  programmatic  decisions  that  result  in: 

•  Cost  growth 

•  Schedule  delays 

•  Performance  shortfalls 


= -  Software  Engineering  Institute  Carnegie  Mellon 


Jim  Smith:  Introduction  to  IMs 
NDIA  systems  Engineering  2009 

©2009  Carnegie  Mellon  University 


Managing  Divergent  Perceptions  -2 


Explicit  versus  tacit/implicit,  “official  truth”  versus  “ground  truth” 

•  Official  truth  is  reflected  explicitly  in  various  policy  statements,  organization 
charts,  program  plans,  directives,  memoranda,  etc. 

•  Frequently  at  odds  with  real  conditions  (e.g.,  actual — versus  ideal — 
programmatic  relationships,  “back  channel”  communications)  that  define  the 
ground  truth 

•  Much  of  this  information  exists  as  tacit  or  implicit  knowledge 

Before  you  can  understand  what  is  actually  happening — and  why — 
underlying  assumptions  and  expectations  must  be  made  explicit 

•  What  is  the  influence?  •  How  effected? 


Between  what  parties? 
For  what  purpose? 


With  what  assurance? 
As  verified  by? 


Software  Engineering  Institute 


Carnegie  Mellon 


Jim  Smith:  Introduction  to  IMs 
NDIA  systems  Engineering  2009 

©2009  Carnegie  Mellon  University 
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Patterns  of  Influence  Relationships 


Patterns  of  influence  relationships  can  provide  indications  of  potential 
risks 

Examples 

•  Cycles,  or  loops 

(e.g.,  “A”  has  a  schedule  dependency  on  “B,”  which  has  a  schedule 
dependency  on  “C,”  which  has  a  schedule  dependency  on  “A”) 

-  Can  lead  to  programmatic  “race  conditions”  because  of  the  delay  between  the  time 
that  an  event  occurs  (e.g.,  delivery  date  delayed  by  rework  to  correct  problems 
discovered  during  testing)  and  when  it  becomes  known  to  other  participants 

•  Hidden — or  indirect — dependencies 

(e.g.,  “A”  has  a  schedule  dependency  on  “B,”  which  has  a  backwards 
compatibility  relationship  with  “C,”  resulting  in  “A”  having  an  indirect 
dependency  on  “C”) 

-  Can  result  in  major  impacts  from  seemingly  unrelated  decisions 
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Construction  and  Use  of  IMs  for  IMA 


IM  Analysis  (IMA)  comprises  four  steps: 

1.  Construct  “strawman”  IMs 

2.  Refine/extend  IMs  during  discovery  process 

-  Create  multiple  node-  and  agreement-centric  IMs  representing  relevant 
stakeholders’  perspectives 

3.  Prepare  composite  IMs,  and  analyze  for  inconsistencies,  gaps, 
clashes,  patterns 

4.  Develop  risk  mitigation  strategies 
Three  types  of  IMs: 

Context-centric:  Provides  a  high-level  overview  of  the  entire  system-of- 
systems  context,  including  all  relevant  participants 

Node-centric:  Represents  influence  relationships,  as  seen  from  the 
perspective  of  a  single  participant 

Agreement-centric:  Provides  detailed  representation  of  an  individual 
influence  relationship,  including  semantics 
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Strawman  I  Ms 


Context-centric  IM  provides  a  high-level,  overview  of  entire  system-of- 
systems  context 

•  Includes  major  participants  and  influence  relationships  that  comprise  the  key 
technical  and  programmatic  drivers 

Strawman  context-centric  IM  based  on  documentation  provided  by 
subject  organization  and  IMA  team  expert  judgment 

•  Serves  as  basis  for  discovery  process 

•  Could  use  outputs  from  a  Critical  Context  Analysis  (CCA)  as  an  input 
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Example:  Strawman  Context-Centric  IM 


Legend:  Strawman 

®  Directed:  A  superior  to  B 

=►{6 j  (e.g,P  reporting 
w  relationship] 

Negotiated:  A  provides 
agreed-upon  service, 
capability,  etc,  to  B  (e.g,( 
“giver-receiver") 
Collaborative:  A  and 
B  agree  to  cooperate 
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Discovery 


Three  goals  of  discovery  process: 

1.  Identify  the  most  critical,  pacing  requirements  that  drive  the  SoS 
context 

2.  To  identify  relevant  internal  and  external  stakeholders  and 
characterize  their  key  concerns,  motivations,  needs,  etc. 

3.  To  develop  context-,  node-,  and  agreement-centric  IMs  that  reflect 
the  “ground  truth”  for  the  relevant  influence  relationships 
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Contextually-Driven  Discovery 


Uses  scenarios  that  relate  to  a  participant’s  context  within  an  SoS  (e.g., 
acquisition  program  office,  operational  tester),  augmented  with  influence 
relationship  templates,  to  structure  participant  interviews 

•  Example:  Your  program’s  budget  has  been  cut  as  a  result  of  a  Congressional 
Committee  “mark.”  How  do  you  evaluate  the  impact  of  this  action  on  your 
ability  to  satisfy  you  program  cost,  schedule,  and  performance  goals?  Who  do 
you  interact  with  in  making  this  determination?  What  information  do  you  need 
to  evaluate?  etc. 

Captures  and  characterizes  assumptions  about  other  stakeholder  roles 
and  responsibilities 
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Templates  Support  Contextually-Driven 
Discovery 


15»ort2i£2L- 


tutfo'*’ 


Documented 


RtWttnftW 


Aids  elicitation  of  influence 
relationships 

•  Different  templates  for  different  SoS 
perspectives  (e.g.,  acquisition 
program  office)  and  contexts  (e.g., 
budget  cut,  schedule  slip) 

•  Lists  typical  classes  of  nodes  with 
which  subject  would  reasonably  be 
expected  to  have  influence 
relationships  (e.g.,  milestone 
decision  authority,  user  community) 

•  Used  to  record  types  of  relationships, 
how  they  are  documented,  their 
status,  etc. 

Serves  as  input  to  generation  of 
“discovery”  I  Ms 

•  Each  row  defines  one  or  more 
influence  relationships 


=  Software  Engineering  Institute 


SoS  Navigator  Update 

Carnegie  Mellon  s  Garcia’  Au9 2008 

fr\  OnfSQ  romQni0  MoMnn  Unix, 


20 


)2008  Carnegie  Mellon  University 


Context-Centric  Influence  Map 


Provides  a  high-level,  overview  of  entire  system-of-systems  context 

•  Includes  major  participants  and  influence  relationships  that  comprise  the  key 
technical  and  programmatic  drivers 

Elaborates/updates  the  strawman  context-centric  IM  based  on 
information  gained  during  interviews 
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Example:  Context-Centric  IM 


Note  greater  detail 
and  additional 
relationships  when 
compared  to 
strawman  context¬ 
centric  IM 


Legend:  Discovery 

®  D i reeled :  A  supe rior  to  B 

(eg.,  reporting 
w  relationship) 

Negotiated :  A  provides 

®  /T\  ag  reed-opon  serv  ice , 

capability,  etc.  to  B  (e.g., 
"giver-receiver") 

_ Collaborative:  A  and 

\_J  B  agree  to  cooperate 


p  rov  ide  s_fgnct»  on 
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Node-Centric  IM 


Represents  a  view  of  immediate  influence  relationships  (i.e.,  not  via  an 
intermediary  agent/agency)  from  the  perspective  of  a  particular 
participant 

Developed  for  each  node  (e.g.,  participant,  organization)  in  a  given  SoS 
context 
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Example:  Node-Centric  IM 
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Agreement-Centric  IM 


Provides  detailed  representation  of  an  individual  influence  relationship 

•  Includes  semantics  of  the  relationship 

•  Example:  what — exactly — does  “delivered”  mean? 

-  Installed  and  ready  to  turn  on? 

-  Or,  shrink-wrapped,  on  a  pallet,  in  some  loading  bay? 

Developed  for  influence  relationships  identified  as  most  critical  to 
success  in  given  SoS  context 
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Example:  Agreement-Centric  IM 


Legend:  Discovery 

Directed:  A  superior  to  B  (e.g., 
V_y  repotting  relationship) 


® — ►© 


Negotiated:  A  provides  atjrssd- 
Lipon  service,  capability,  etc.  to  B 
(e.g.,  "givet-ieceiverl 


©Collaborative:  A  and  B 
agree  to  cooperate 


Agreement; 

•  deliver_no__earIier_than 

*  delivery o Jater J han 


•  agreed  functionality 

•  agreed_qualily_attribules 

•  agreed_assuranc© 

-requirements_dependency- 


Receiver: 

*  earliestneeddate 

*  latest_need_date 

•  min_acceptable_functionality 
■  desiredfu  notion  ality 

•  required_quality_aNributes 


Giver: 

•  off©  red_de  I  ive  ry_d  ate 

•  offered_fij  notion  a  lity 

•  off©  red_q  ua  I  ity_attri  b  ute  s 
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Analysis 


Identify  potential  influence  interrelationship  risks 
Characteristics  of  analysis  approach 

•  IMA  team  compares  the  IMs  developed  during  the  preparation  and  discovery 
steps  to  identify  and  characterize 

-  Differences  with  respect  to  the  strawman  IMs 

-  Differences  between  participants’  perspectives  of  a  given  influence 
relationship 

-  New,  changed,  deleted,  or  missing  influence  relationships 

•  Analysis  is  performed  from  3  perspectives 

-  Context-centric 

-  Node-centric 

-  Agreement-centric 

•  IMA  team  members  use  identified  patterns  of  influence  relationships 
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Context-Centric  IM  -  Analysis  -1 


Provides  a  composite  of  top-down  context-centric  IM  developed  during 
interviews,  overlaid  with  individual  node-centric  IMs 

Highlights  divergent  perceptions  of  influence  relationships  obtained 
during  interviews 

•  New,  or  “discovered”  influence  relationships  (i.e.,  influence  relations  not 
apparent  from  context-centric,  top-down  perspective) 

•  Deleted  influence  relationships,  that  appear  in  a  context-centric  view,  but  not 
at  the  node-centric  perspective 

•  Conflicted  influence  relationships,  for  which  different  participants  have 
divergent  interpretations 

Provides  input  to  node  IM  -  analysis  and  risk  mitigation  planning 
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Context-Centric  IM  -  Analysis  -2 


Three-step  process  for  construction  of  “analysis”  IMs: 

1 .  “Normalize”  the  IMs  developed  during  discovery  phase 

-  Common  naming  scheme  (i.e.,  resolve  synonyms/homonyms) 

-  “Apples-to-apples”  comparison  of  relationship  characteristics  (e.g., 
schedule-to-schedule,  functionality  desired-versus-promised) 

2.  Prepare  a  composite  IM  at  the  appropriate  level  (i.e.,  context,  node,  or 
agreement) 

-  Overlay  different  stakeholders’  views 


How  “A”  sees  the 
relationship 


How  “B”  sees  the 
relationship 


Composite  view  of 
the  relationship 
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Context-Centric  IM  -  Analysis  -3 


Three-step  process  for  construction  of  “analysis”  IMs  (continued): 

3.  Examine  the  resulting  composite  IMs,  highlighting  any  omissions,  conflicts, 
or  differences  of  interpretation  on  a  relationship-by-relationship  basis.  For 
example: 

-  Do  both  stakeholders  participating  in  a  given  relationship  “see”  the  same 
thing?  For  example,  do  both  parties  in  a  “giver-receiver”  relationship  agree 
on  the  delivery  date?  What  they  even  mean  by  “delivered”?  The  required 
functionality?  How  that  functionality  will  be  assured? 

-  Does  only  one  stakeholder  perceive  the  existence  of  a  relationship?  Are 
the  stakeholders  “talking  past  each  other”? 

•  Does  one  stakeholder  perceive  the  relationship  as  a  schedule 
dependency,  while  the  other  one  sees  a  backwards-compatibility 
relationship? 

•  They  could  both  be  referring  to  the  same  relationship,  but  their 
respective  reference  frames  could  prevent  them  from  realizing  that 
these  relationships  are — in  fact — the  same 
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Example:  Context-Centric  IM  -  Analysis 


Legend:  Analysis 


©=KD 


Directed:  A  superior  to  B  (e  g., 
reporting  relationship) 


®  Negotiated:  A  provides  agreed- 

►:  B  )  upon  service,  capability,  etc.  to  B 
(e.g.,  “giver-receiver") 


® - ® 


Collaborative:  A  and  B 
agree  to  cooperate 

Added  during  Discovery 
<Deleted  during  Discovery> 

[Conflict  detected  during  Discovery] 


reporting 


Background 

Knowledge 


reporting 

X 


reporting 


6 


cost_plus_contract 


reporting 

\ 

requirements_dependency 

1 


requirements_dependency 


[schedule_dependency] 


requirements 

cost  nl"c  ~ontr^t 

schedu»e,dependencyrequ.rements 


/[,«,„ . 

n 


requirements_dependency 

[schedule_dependency] 


scheduledependency 


' 


requirements 


requirements 

V  cost_plus_contract 


provides_function 


builds 

T 


provides_function 


builds 

r 


providesfu  notion,. 


provides_function 


provides_f  unction- 
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Node-Centric  IM  -  Analysis 


Characterize  divergent  perceptions  of  influence  relationships 

•  Refinement  of  the  context  IM  -  analysis 

•  Captures/identifies  top-level  changes,  additions,  deletions,  and  conflicts  for 
relevant  influence  relationships 

•  Supports  prioritization  of  relationships  requiring  detailed  analysis  at 
agreement-centric  level 

Provides  inputs  to  risk  mitigation  planning 
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Example:  Node-Centric  IM  -  Analysis 


Legend:  Analysis 


®— ® 
® — ►© 
® - ® 


Directed:  A  superior  to  B 
(e  g,,  reporting  relationship) 
Negotiated:  A  provides 
agreed-upon  service, 
capability,  etc,  to  B  (e,g., 
"giver-receiver") 
Collaborative:  A  and  B 
agree  to  cooperate 
Added  during 
Discovery 
^Deleted  during 
Discovery* 

[Conflict  detected  during 
Discovery] 
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Agreement-Centric  IM  -  Analysis 


Captures  detailed  enumeration  of  changes,  additions,  deletions,  and 
conflicts  at  the  individual  agreement  level 

•  How  has  an  agreement  changed? 

•  What  is  the  impact  of  that  change? 

Provides  input  to  risk  mitigation  planning 
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Example:  Agreement-Centric  IM 


Legend:  Analysis 

Dlr&ct&d :  A  superior  to  B  (e.g . , 

_ 

reporting  relationship) 

Negotiated:  A  provides  agreed- 
upon  service,  capability,  etc.  to  B 
(a.g.,  tglwer-recalvern) 

Collaborative-  A  and  B 

Ky 

/[T\ 

w 

agree  to  cooperate 

A  tided  during  Discovery 

< Deleted  during  Dlecovory> 

[Conflict  detected  during  Discovery] 

— 

—  - 

Agreement: 

•  deliver_no_earIier_than 

•  [deliver_no_later_than] 

■  ag  reed  f  u  action  a  I  i  ty 

•  [agre©d_quality_attribut©s] 

•  [agreed_assuranoe] 


requirements_dependency] 


Receiver: 

•  earliestneeddate 

•  [latest_need_date] 

•  miri_acceptable_f<j  rationality 

•  desired_functionality 

•  [  req  u  i  redqu  a  litya  (tributes] 

•  required_ass  ura  nee 


Analysis 


Giver: 

*  [offered_dei i very _d ate] 

*  offered_functionality 

*  [offered  quality  attributes] 

*  coffered  assurance> 
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Mitigation  Planning 


Mitigation  strategy  and  plans  developed  for  prioritized  risks  identified 
during  analysis  phase 

•  Develop  and  implement  mitigation  strategy  and  plans  in  facilitated  workshop 

•  Monitor  for  any  changes 

•  Maintain  and  update  IMs 
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Example:  Mitigation  of  Requirements  Risk 
Identified  in  Analysis  Agreement-Centric  IM 


Analysis  identified  changes  in  the 
requirements  dependency  between  “P” 
and  “B” 

•  Changes  to  “need”  and  “delivery”  dates,  and 
desired/offered  quality  attributes 

•  “B”  has  articulated  new  requirements  for 
assurance,  while  “P”  has  dropped  some 
previously-offered  assurances 

Analysis  provides  a  basis  for  “B”  and  “P” 
to  negotiate  a  new  agreement — or  identify 
that  no  agreement  is  possible 


Agreement: 

*  deliver_no_earlier_than 

*  [delivernolaterthan] 

*  agreed_fu  nationality 

*  [agreed_qua!ity_attributes] 

*  [agreed_assu ranee] 

h — [re  q  u  i  rements_de  pendency] 


Receiver: 

*  earl  iest=ineed_d  ate 

*  [latest_need_date] 

*  min_acceptable_functionality 

*  desired_fu  nationality 

*  [required  quality  attributes] 

*  required_assurance 


Giver: 

•  [offered  delivery  date] 

•  offeredfu  nationality 

•  [offered_quality_attributes] 

•  <offered_assurance> 


Identified  aspects  of  the  agreement — 
which  may  have  been  previously 
unstated —  that  need  to  be  watched  for 
future  changes  (e.g.,  quality  attributes) 
based  on  their  potential  to  affect  cost, 
schedule,  or  performance 


Legend:  Analysis 

Directed:  Asuperiorto  B  (e.g., 

\C/  ^  VI/  reporting  relationship) 

Negotiated:  A  provides  agreed- 

/'a') 

(e.g.,  Jgrver-reoeiver) 

Collaborative:  A  and  B 

agree  to  cooperate 

— 

—  • 

Ajded  during  Discovery 

<Deleted  during  Discovery> 

[Conlict detected  during  Discovery] 
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Key  Points 


Conventional  governance  and  acquisition  techniques  and  processes 
provide  an  incomplete,  and  often  incorrect,  understanding  of  how  the 
dynamics  of  systems  of  systems  bear  on  the  eventual  success  or  failure 
of  the  enterprise 

I  Ms — and  associated  IMA  method: 

•  Permit  identification  of  disconnects  between  stakeholder  perspectives  of 
influence  interrelationships,  and  deviations  from  “official  truth” 

•  Is  useful  for  any  organization  involved  in  systems  of  systems  with  multiple 
stakeholders,  and  conflicting  goals 

-  Particularly  relevant  for  program  managers,  senior  executives,  and  policy  makers 

•  Provides  sufficient  detail  for  focused  mitigation  actions 
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NDIA  systems  Engineering  2009 

©2009  Carnegie  Mellon  University 
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Using  Model  Based  Systems  Development 
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Sanford  Friedenthal 
Lockheed  Martin 
sanford.friedenthal@lmco.com 


Topics 


7 


■  Model-based  Systems  Development  Motivation, 
Scope,  and  Challenges 

■  MBSD  Approach  Using  System  Architecture  Model 
as  Integration  Framework 

■  MBSD  Observations 

■  INCOSE  MBSE  Initiative 

■  Summary 


MBSD  Motivation,  Scope, 
and  Challenges 


SE  Practices  for  Describing  Systems 


Future 


Moving  from  Document  centric  to  Model  centric 


Model-based  Systems 
Development  (MBSD) 


Life  Cycle  Support 


Formalizes  the  practice  of 
systems  development 
through  use  of  models 
Broad  in  scope 

-  Integrates  with  multiple 
modeling  domains  across 
life  cycle  from  system  of 
systems  to  component 

Results  in  quality/productivity 
improvements  &  lower  risk 

-  Rigor  and  precision 

-  Communications  among 
system/project  stakeholders 

-  Management  of  complexity 
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MBSD  Must  Integrate 
across  Modeling  Domains 


Software 

Design 


Engineering 

Analysis 


Hardware  Human  System 
Design  Integration 
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Modeling  Challenges 


■  Lots  of  good  modeling  going  on,  but: 

-  Modeling  practices  in  people’s  head, 
and  not  well  codified  and  shared 

-  Modeling  still  done  in  stovepipes,  and 
not  fully  integrated  into  systems 
development  workflow 
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MBSD  Approach  Using 
System  Architecture  Model 
as  Integration  Framework 


Using  System  Architecture  Model 
as  an  Integration  Framework 


®fta©Bs 
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Using  the  System  Architecture  Model 
to  Flowdown  Reauirements 


B-Spec 


Component  Design 
&  Implementation  Level 


SW 

Comp  1 
Spec 

SW 

Comp  X 
Spec 


Trade  Studies, 
Simulation, 

Specification  Reviews, 
etc. 


L_ 


4L 


Behavior, 
^Structure  & 
Requirements 


(from  John  Watson/LMC 
SysML  Info  Days  presentation) 


©  Copyright  Lockheed  Martin  Corporation  All  Rights  Reserved 


System  Decomposition  Process  using  SysML 

? 


Analyze  System  Level  Requirements 


Analyze  System  Services 


Identify  the  Subsystem 


Incorporate  Additional 
Analysis  as  Needed 


Derive  and  Allocate  Requirements 
to  Subsystem 


Continue?! 


Complete  Subsystem  Specs 


Input 


•pH 

CucD 

.  ^  | 

m- 


Analyze  Subsystem  Collaboration  to 
Satisfy  the  System  Services 


£ 


Trade  Studies,  R&D, 
Simulation,  Specification 
Reviews,  etc. 


The  Subsystem  shall 


<5S> 


(from  John  Watson/LMC 
SysML  Info  Days  presentation) 


System  Architecture  Model  to 
Support  Tradeoff  Analysis 


System  Architecture  Model 


Analysis 
W  Results 


_ )\ 

nqi 

\ 

\_M  > 

r  ! 

Subsystem 

Alternativel 

Alternative2 

Alternatives 

Sensor 

Sensorl 

Sensor2 

Sensor3 

Processor 

Processorl 

Processor2 

Processors 

Control 

Controll 

Control 

Control3 

Criteria 

Weight 

Alt  1 

Alt  2 

Alt  3 

Performance 

0.5 

7 

5 

5 

Reliability 

0.2 

4 

6 

5 

Cost 

0.3 

3 

5 

8 

Effectiveness 

5.2 

4.2 

5.9 

Reliability 
Performance 


par  Overall  Effectiveness 


Effectiveness 


Optimization 


{E  =  Sum  [w1*u1(P)  +  w2*u2(R)  +w3*u3©]} 


Objective  Function 
PRC 

n _ □ _ n 


T~ 

Peformance 

Reliability 

Cost 
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Requirements  Management 


Typical  Integrated  Tool 
Environment 


Project  Management 


SoS/Enterprise  Modeling 
UPDM 


System  Modeling 
SysML 


Software  Modeling 
UML  2.0 


Hardware  Modeling 
VHDL,  CAD,  .. 


Simulation  &  Visualization 


Assess  the 
state  of  your 
practice 


CODIFY 

Codify  the 
practice 


I  nfrastructure  &  Su 


Practices 
Tools  &  Testbed; 
Training 


Deploying  MBSD  as  part  of 
Improvement  Process 

ASSESS 


PILOT 

Pilot  the 
practice  and 
tailor  the 
approach 


DEPLOY 

I  ncrementally 
integrate 
changes  into 
the  current 
workflow 


Plan  the 
improvement 


PLAN 
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Observations  and  Summary 


MBSE  Observations 


7 


■  Transition  from  document-centric  to  model-centric 
is  a  cultural  change 

■  Well  defined  MBSE  method  is  essential 

■  Multiple  tool  vendors  provide  a  range  of  price  point, 
capability,  and  standards  conformance 

■  MBSE  training  should  include  language,  method, 
and  tools 

■  Employ  pilots  to  validate  your  MBSE  approach 

■  Need  buy-in  from  program  and  customer  on  MBSE 
benefits,  approach  and  deliverables 

■  Scope  model  to  support  program  objectives  and 
within  program  constraints 

■  A  lot  has  been  learned,  but  much  more  remains 


©  Copyright  Lockheed  Martin  Corporation  All  Rights  Reserved 


INCOSE  MBSE  Initiative 


INCOSE  MBSE  Initiative  Charter 


7 


■  Promote,  advance,  and  institutionalize  the  practice  of 
MBSE  to  attain  the  MBSE  2020  Vision  through  broad 
industry  and  academic  involvement  in: 

-  Research 

-  Standards 

-  Processes,  Practices,  &  Methods 

-  Tools  &  Technology 

-  Outreach,  Training  &  Education 


INCOSE  MBSE  Roadmap 


2010 


2020 


2025 


Summary  - ~7^ 

■  MBSE  is  a  key  practice  to  advance  complex  systems 
development 

■  Standards  such  as  SysML  are  critical  enablers  of  MBSE 

■  Multiple  tool  vendors  implementing  the  standard 

■  System  architecture  model  and  standards  based 
approach  facilitate  Integration  across  modeling  domains 

■  Growing  interest  and  application  of  MBSE 

■  INCOSE  MBSE  helping  to  advance  and  promote  MBSE 


TECHNOLOGY  DRIVEN. 


WARFIGHTER  FOCUSED. 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


A  Systems  Engineering  Model  for  Roadmap  Alignment 

Presented  by:  Si  Dok 
Prepared  by: 

Si  Dok,  Harsha  Desai,  and  John  Fitch 


October  2009 


INTRODUCTION 


1 .  Discuss  problem  space 

2.  Discuss  condition  of  problem  space 

3.  Discuss  affinity  process 

4.  Discuss  architectural  function 

5.  Roadmap  alignment 
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DECOMJ)  BREAKDOWN  OF  THE  PROBLEM  SPACE 


Things  evolve  at  their  own  rate! 


Information  Management 


Knowledge  Engineering 
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What  does  the  customer  say  they  want  for  now? 


Things  do  not  align  themselves! 
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TARDEC  PROBLEM  -  CURRENT  SCENARIO 


Things  do  not  align  themselves! 


Combat 


Tactical 


Brigade 

Combat 


ft 

■ 
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DEVELOPING  THE  FRAMEWORK 


RDECOM 


OUR  STRUCTURED  APPROACH  LEADS  TO 

STABILIZATION 


Se 


FOC 

~  ......  context 

Capabilities  «* - 

(High  Level) 


"delta”  capabilities 


Portfolio 

mapping 

◄ - 


Platform 

context 


Capability  Gaps 
(Voice  of 
customer) 


Technology  Links 


Solutions 

Portfolio 


Investments 


DATA 

DRIVEN! 


Platforms 


Programs 
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TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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Market  “Pull” . Technology  “Push” 


War  fighter 
Needs 

Batch/Event 

Technology  Focus 
Areas 

Capability  Gaps 

Driven 

•Power  and  Energy 

•Prioritize 

•Mobility  &  Logistics 

•Identify  Common 
•Decompose 

•Lethality 

Function 

KPP 

SWaP 

Platform 

Cost 


Requirements 

“Model” 

•Functional 
•Performance 
•Platform  Constraints 


Continuous 
Ownership  model 


4 


ALIGN 


Function 
KPP 
>  SWaP 
Platform 
Cost 


Technology 

Roadmap 

•Functional 
•Performance 
•Platform  Constraints 
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ROADMAP  CONCEPT 


Decisions: 


D1:  What  is  our  vehicle/platform  roadmap?  Which  capability  gaps  will  be  filled  in  which 
increments  (customer) 

D2:  What  capability  gaps  will  TARDEC  attempt  to  fill?  With  which  technologies  and 
solution  sets?  (TARDEC) 

D3:  How  will  we  integrate  our  technology  enablers  into  high-value  (multi-use)  solution 
sets?  (TARDEC) 

D4:  What  portfolio  of  programs  (investments)  will  best  deliver  the  technologies  and 
solutions  that  meet  the  warfighters’  needs?  (TARDEC) 


Gather  summaries  of  existing  ATOs, 
SBIRs,  Congressional  Adds  and 
identify  their  deliverables  in  terms  of 
technology  maturation  (KPPs  +  TRLs) 


Sol  1 
KPP1 
SWaP 
TRL 

Sol  2 
KPP  1 
SWaP 
TRL 


Solutions 


i 


Wire  together  technology 
enablers  to  form  solution  sets 


D3 

Technologies 

Tech  1 
KPP  1 
KPP  2 
TRL 

Tech  2 
KPP  1 
KPP  2 
TRL 


Gather  and  refine  existing 
roadmaps  +  technologies  ideas 
from  SMEs 


Platforms 


Veh  X 
SWaP 


I  Gather  existing  Vehicle 
roadmaps,  capabilities 
Veh  y  allocated  to  increments 

swaP^^d  and  SWaP  constraints 

Cap’s 


D1 


Capabilities 

Cap  A 
KPP  1 
KPP  2 

KPP  3  | 

Cap  B 
KPP  1 
KPP  2 


Identify  opportunities  by  analyzing  PM  1-N  lists, 
TFT  Tech  Gaps  and  WFOs 

Decompose  capability  gaps  into  functions  and 
KPPs  using  source  requirements  documents 
(due  diligence  to  confirm  alignment) 
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1 .  Everyone  will  fall  into  one  of  the  alignment  realms 

2.  Using  non-conventional  SE  processes 

3.  Anything  can  be  aligned 

4.  Structured  information  model  -  ready-for-use  pattern 
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Q&A  Session 
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Informs 

S&T 

Portfolio 

Decisions 


Maintain 

Warfighter 

Needs 


Plan  capability  evolution 

Identify  capabilities  gaps 

Prioritize  capabilities  gaps 

Time-align  capabilities  and 
enabling  technologies 

Identify  common  needs 

Performance-align  capabilities 
and  enabling  technologies 

Refine  capabilities  needs 

Maintain 
RDECOM 
technology/soluti 
on  taxonomy 


Plan  technology  evolution 
Identify  technology  investments 


Indentify  technology  dependencies 


Maintain 

platforms 

taxonomy 


Capture  S&T 

program 

portfolio 


y 

Allocate  capabilities  to 

\ 

platform  upgrades 

Plan  programs 

Re-plan  programs 

Data  Elements 


CAPABILITY 


VOICE  OF 
CUSTOMER 


PLATFORM 


TECHNOLOGY 


PROGRAMS 
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Informs 

S&T 

Portfolio 

Decisions 


Maintain  RDECOM 
technology/solution 
taxonomy 


Provide  context  for  capability  needs 


Plan  capability  evolution 


Identify  capabilities  gaps 


Prioritize  capabilities  gaps 


Identify  enabling  technologies 


Identify  common  needs 


Time-align  capabilities  and  enabling 
technologies 


Performance-align  capabilities  and  enabling 
technologies _ 


Refine  capabilities  needs 


Plan  technology  evolution 


Identify  technology  investments 


Time-align  technologies  and  prerequisite 
programs 


Performance-align  technologies  and 
prerequisite  programs 


Maintain  platforms 
taxonomy 


Plan  platform  evolution 


Allocate  capabilities  to 
platform  upgrades 


Capture  S&T 
program  portfolio 


Plan  programs 


Re-plan  programs 


Track  program  status 


Data  Elements 


CAPABILITY 


VOICE  OF 
CUSTOMER 


PLATFORM 


TECHNOLOGY 


PROGRAMS 
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FUNCTIONAL  ARCHITECTURE 


Informs 

S&T 

Portfolio 

Decisions 


Maintain 
RDECOM 
technology/sol 
ution  taxonomy 


Maintain  platforms 
taxonomy 


Capture  S&T 
program  portfolio 


Provide  context  for  capability  needs 


Plan  capability  evolution 


Identify  capabilities  gaps 


Prioritize  capabilities  gaps 


Identify  enabling  technologies 


Identify  common  needs 


Refine  capabilities  needs 


\\ 


Plan  technology  evolution 


Identify  technology 
investments 


ndentify  technology 


Plan  platform  evolution 


Allocate  capabilities  to  platform 
upgrades _ 


Plan  programs 


Re-plan  programs 


Data  Elements 


Time-align  capabilities  and 
enabling  technologies 

Performance-align  capabilities 
and  enabling  technologies 

CAPABILITY 

VOICE  OF 
CUSTOMER 

prerequisite  programs 
Performance-align  technologies 
and  prerequisite  programs 


Time-align  interdependent 
technologies 


Performance-align  interdependent 
technologies 


PLATFORM 


TECHNOLOGY 


PROGRAMS 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

17 


Informs 

S&T 

Portfolio 

Decisions 


Data  Elements 


Time-align  capabilities  and  enabling 
technologies _ 

Performance-align  capabilities  and  enabling 
technologies _ 


Time-align  technologies  and 
prerequisite  programs 

Performance-align  technologies 
and  prerequisite  programs 

PROGRAMS 


Maintain  platforms 
taxonomy 


Capture  S&T 

program 

portfolio 


Plan  programs 


Re-plan  programs 


Track  program  status 


CAPABILITY 


VOICE  OF 
CUSTOMER 


PLATFORM 


TECHNOLOGY 


Provide  context  for  capability  needs 


Plan  capability  evolution 


Identify  capabilities  gaps 


Prioritize  capabilities  gaps 


Identify  enabling  technologies 


Identify  common  needs 


Maintain  RDECOM 
technology/solution 
taxonomy 


Refine  capabilities  needs 


Plan  technology  evolution 


Identify  technology 
investments 


Indentify  technology  dependencies 
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Overview  of  Department  of  Defense 
(DoD)  Software  Engineering  Initiatives 

Mr.  Scott  Lucero 

Deputy  Director,  Software  Engineering 

Systems  Engineering  Directorate 
Office  of  the  Director,  Defense  Research  and  Engineering 

12th  Annual  NDIA  Systems  Engineering  Conference 

October  29,  2009 
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UNCLASSIFIED 


Elements  of  a  DoD  Strategy  for 
Software  Engineering 


•  Support  Acquisition  Success 

-  Ensure  effective  and  efficient  software  solutions  across  the 
acquisition  spectrum  of  systems,  SoS  and  capability  portfolios 

•  Improve  the  State-of-the-Practice  of  Software 
Engineering 

-  Advocate  and  lead  software  initiatives  to  improve  the  state-of-the- 
practices  through  transition  of  tools,  techniques,  etc. 

•  Leadership,  Outreach  and  Advocacy 

-  Implement  at  Department  and  National  levels,  a  strategic  plan  for 
meeting  Defense  software  requirements 

•  Foster  Software  Resources  to  meet  DoD  needs 

-  Enable  the  US  and  global  capability  to  meet  Department  software 
needs,  in  an  assured  and  responsive  manner 


Promote  World-Class  Leadership  for  Defense  Software  Engineering 
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NDIA  Top  Software  Issues 
September  2006 


1 .  The  impact  of  requirements  upon  software  is  not  consistently  quantified 
and  managed  in  development  or  sustainment.  “SW  Requirements” 

2.  Fundamental  system  engineering  decisions  are  made  without  full 
participation  of  software  engineering.  “SE/SW  Integration” 

3.  Software  life-cycle  planning  and  management  by  acquirers  and 
suppliers  is  ineffective.  “SW  Sustainment” 

4.  The  quantity  and  quality  of  software  engineering  expertise  is 
insufficient  to  meet  the  demands  of  government  and  defense  industry. 

“Human  Capital” 

5.  Traditional  software  verification  techniques  are  costly  and  ineffective 
for  dealing  with  the  scale  and  complexity  of  modern  systems. 

“SW  Testing” 

6.  There  is  a  failure  to  assure  correct,  predictable,  safe,  secure  execution 
of  complex  software  in  distributed  environments.  “SW  Assurance” 

7.  Inadequate  attention  is  given  to  total  lifecycle  issues  for  COTS/NDI 
impacts  on  lifecycle  cost  and  risk.  “SW  COTS  /  NDI  /  Reuse” 
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Top  Software  Issues  -  2006  vs. 
Software  Systemic  Findings  -  2008 


National  Defense  Industrial  Association  (NDIA) 
Top  7  Software  Issues 
September  2006 


ODDRE/SE  Systemic  Analysis  of 
Program  Support  Review  Findings 


Software  Human  Capital 


Software  Human  Capital 


0  Resources 
0  Quality  Level 


Software  Requirements 


Software  Requirements 


0  Engineering 
0  Management 
0  Acquisition  Strategy 


Systems/Software 

Integration 


Systems/Software 

Integration 


0  Systems  of  Systems 
0  Interoperability 
0  Tech  Refresh 


Software  Assurance 


Software  Assurance 


Software  Development 


□  Software  Testing* 

□  Software  Sustainment/Maintenance* 
I  Software  COTS/NDI* 

D  Technology  Readiness 

□  Software  Architecture 


Software  Metrics 


□  Software  Metrics 

□  EVM 


Software  Engineering 
Management 


0  Project  Planning 
0  Management  Oversight 
□  Software  Configuration  Management 


Knowledge  Sharing 


□  Process 

□  Reporting 
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Current  Software  Engineering 

Initiatives 


•  Program  Support 

-  Provide  software  support  for  acquisition  program  reviews. 

Develop  independent  schedule  and  defect  estimates. 

*  Human  Capital 

-  Software  Acquisition  Training  and  Education  Workgroup: 

Establish  SW  competencies  across  the  acquisition  career  fields 

-  Reference  Curriculum  for  Graduate  Study  of  Software  Engineering: 
Version  1.0  completed  this  month,  to  be  sustained  by  IEEE  and  ACM. 

•  Advance  the  State  of  the  Practice 

-  Software  Sustainment,  NDIA  Software  T&E  Summit/Workshop 

*  Policy  and  Guidance 

-  Earned  Value  Management,  Military  Handbook  for  Work  Breakdown 
Structures:  MIL-HDBK-881. 

-  Oversight  of  Services’  SW  Acquisition  Process  Improvement 
Programs. 
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Notional  Example  of 
Schedule  Feasibility  Analysis 


Current  Plan  (Dec  2013) 

I 


k  Profile:  Hid  Date 
>0  solutions  sampled) 


_ 


'\  r 


Review  Team  Projection 
(50th  percentile  -  Dec  2014) 


End  Date 

-^T/26/2014 

3/17/2014 
5/6/2014 
6/25/2014 
8/14/2014 
10/3/2014 
J 1/22/2014 
"^1/1 1/2015 
-(3/2/2015 
21/2015 
10/2015 
30/2015 
"9718/2015 
11/7/2015 
12/27/2015 


Cum  Probability  (%) 

1 

3 

8 

13 

22 

31 

41 

54 

65 

75 

83 

90 

95 

99 

100 


Likelihood  of  delivery  to  current 
schedule:  less  than  1% 


Risk  Profile:  End  Date 

(1000  solutions  sampled) 


Review  Team  Projection 
(50th  percentile  Dec  2014) 


Current  Plan 
(Dec  2013) 


O 

c 

3 

“0 


12/7/2013 


J  \ 


6/25/204  1/11/2015 

End  Date 


7/30/2015 
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hri  Software  and  Systems  Reliability 


*  DoD  has  renewed  emphasis  on  systems  reliability  and 
lifecycle  costs  of  shortfalls 

-  DDRE  effort  underway  to  consolidate  software  reliability  guidance 

•  Starting  to  use  parametric  models  to  project  numbers  of 
latent  software  defects  and  discovery  rates 


-  Used  to  support: 

-  Development  of  satellite  launch  plans 


Software  Block  Maturity  -  Nov  2008 

-  Aircraft  production  decisions 

-  Operational  test  readiness  reviews 

1  2.00  - 

Gauging  software  reliability  1 150 

using  Mean  Time  to  Defect  f 

(MTTD)  discovery  % 

0.50  - 
1 

j 

0.00  - 

£ 

q  2  ^  q  3  1 1  3.2  3.3  3.4  3.5.3 

k  A  A  A  A 

| - Target  Maturity - Actual  Maturity  A  Build - Linear  (Actual  Maturity)  | 

System  Testing  DRs  /  Test  Hr 
(DT  events  to  IOT&E) 
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Software  Human  Capital  Efforts 


Software  Industrial  Base  Study  -  July  2007 
There  is  a  choke-point  in  availability  of  top-tier  software 
managers,  architects,  and  domain  experts. 

Supply  of  sufficiently  trained  SW  developers  is  inadequate  near-term. 


•  Software  Acquisition  Training  and  Education  (SATEWG) 

-  Chartered  February  2008  by  USD(AT&L)  to  add  software  competencies  to 
DoD’s  13  acquisition  career  fields 

-  Recent  accomplishments: 

-  Developed  software  competency  framework, 

-  Established  SPRDE  software  competencies 

-  Gap  analysis  of  SATEWG  competency  framework  and  DAU’s 
Software  Acquisition  Management  courses 

-  Current  focus  is  on  PM,  Contracting  and  Test  career  fields 

•  Graduate  Software  Engineering  Reference  Curriculum  (GSwERC) 

-  Partnership  with  Industry  and  Academia 

-  Version  1 .0  completed  September  2009 

-  Transitioned  to  IEEE  and  ACM  for  sustainment 
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Software  Sustainment  Challenges 


*  Software  intensive  systems  encourage*: 

-  Build-a-little,  test-a-little,  field-a-little  risk  reduction 

-  Incremental  and  spiral  development  efforts 

-  Concurrent  planning,  development  and  sustainment  activities 

*  No  longer  a  natural  ‘break  point’  where  software 
development  can  be  transitioned  to  a 
sustainment  organization 

-  Technical  capability  of  Government  sustainment  organizations 
reduced  due  to  acquisition  reform 

*  Planning  for  software  sustainment  now  a  lost  art 

-  Acquisition  programs  no  longer  produce  MIL-HDBK-347  Computer 
Resource  Life  Cycle  Management  Plans 


Better  planning  needed  to  partition  software  work 
among  multiple  developers  and  increase  competition 
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*AIAA  Infotech  Conference  2009,  Software  Sustainment  Challenges  in  Defense  Acquisition 


NDIA  Software  Test  and  Evaluation 
Summit/Workshop  -  Sep  2009 


*  Purpose:  “Recommend  policy  and  guidance 
changes  to  emphasize  robust  software  T&E 
approaches  in  Defense  acquisition.” 

*  Speakers  from  Government,  Industry  and 
Academia 

*  Conducted  workshops  on: 

-  How  much  software  T&E  is  enough 

-  Software  T&E  involvement  across  the  lifecycle 

-  Emerging  paradigms:  SOA,  SoS,  Security 

*  Workshops  specifically  addressed: 

-  Policy  &  guidance,  Human  capital,  RFP  language, 

SW  T&E  tools 

*  NDIA  Software  Experts  and  DT&E  sub-committee 
to  produce  white  paper  by  December  2009 
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Software  Measurement  and  Analysis 

Improvement  Areas 


Determine  better 
methods  of 
obtaining  cost 
estimating  data 


Generate  software 
appropriate  WBS 


Improve  estimation 
tools,  techniques,  & 
practices 


Find  best  Earned  Value 
Management  (EVM) 
practices  for  SW 


Link  quality  indicators 
to  EVM 


Concepts  -  Requirements  -  Arch/Design  -  Development  -  Maintenance 


Integrate  software  guidance  into  proven 
management  techniques 
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Software  Earned  Value 
Management  (EVM)  Study/Pilot 


*  Develop  methods  to  combine  EVM  and  software 
metrics  to  predict  cost  and  schedule  overruns 

*  Piloted  on  a  5-year  ACAT  1 D  software 
development  program 

*  Pilot  indicator  shows  estimate-at-completion 
(EAC)  forecasts  for: 

-  Existing  program  management  plans 

-  Milestone-based  EVM  measures 

-  Software  metrics,  i.e,  growth  profile  of  size,  effort,  defects 


Equivalent  EAC  forecasts  provide  an  increased 

confidence  in  project  plans 
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DOLLARS  ($K 


Estimates  at  Completion  (EAC)  for 
Metrics,  Earned  Value,  Program  Plans 


SW  Metric 
EAC 


Spiral  -  Sample  Summary 
s  Measurement  Baseline 


'BAC  =  $1420 


$i \A 

J  $1,559  1 

,  $1,579  J 

Program 

Plan 

Completion 


$1^45 . 


$1,051  / 


$941 


$792 

^^$1,032 

/ 

$706 

!5 

$47^^ 

i  $565 

5338 

Data  Date 

EV  Metrics 

Calculated  Schedule  Range: 
09/08/08-  10/12/08 

Calculated  Estimate  at  Complete 
Range: 

$1,797K-  $2,51 5K 
Cumulative  CPI  =  0.79 


SW  Metrics 

Calculated  Schedule  Range: 
05/14/08  -  06/18/08 

Calculated  Estimate  at  Complete 
Range: 


Oct07  Nov07  Dec07  Jan08  Feb08  Mar08  Apr08  May08  Jun08  Jul08  Aug08  Sep08  Oct08  Nov08  Dec08 

. _ _ 

- BCWS(PV)  - BCWP(EV)  - ACWP(AC) 


- BAC 


- ETC 


Confidence  increases  as  EACs  overlap 
Multiple  measures  reaching  the  same  conclusion 
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Questions/Discussion 


Contact  Information: 

Don  Scott  Lucero 

Deputy  Director,  Software  Engineering 
Systems  Engineering  Directorate,  Defense  Research  and  Engineering 

Scott.Lucero@osd.mil 
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Outline 


Raytheon 

Missile  Systems 


■  Introduction  of  Model-driven  Engineering  (MDE) 

■  History  of  MDE  at  Raytheon  Missile  Systems 

■  Intentions  of  Using  MDE  for  Integrated  Flight  Simulation  (IFS) 
Development 

■  MDE  Tool  History  Example 

■  Model  Lifecycle  Comparison 

■  MDE  Process  Flow 

■  Time  Savings  Comparison 

■  Performance  in  Integrated  Flight  Simulations 

■  Common  Pitfalls 

■  Conclusions 
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Introduction  of  Model-driven 
Engineering 


Raytheon 

Missile  Systems 


■  Model-driven  Engineering 

-A.K.A.  Model-driven  Development  (MDD) 

-  Software  development  methodology  that  focuses  on  creating  models 
rather  than  algorithms 

-  Domain  experts  maintain  more  control  of  the  software  end  product 

-  Promotes  compatibility  and  communication  between  individuals/teams 

■  One  Tool’s  Role  in  MDE 

-  Simulink®  is  a  Popular  tool  for  domain  experts’  development  of  system 
models 

-  Real-Time  Workshop®  Embedded  Coder  provides  MDE  interface  to 
Integrated  Flight  Simulations  (IFSs)  through  automatic  generation  of 
C/C++  code 

-  IFS  engineer  owns  process  of  creating  code 


Real-Time  Workshop®  provides  an  MDE  interface  to  the  IFS 
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History  of  MDE  at  Raytheon 
Missile  Systems 


Raytheon 

Missile  Systems 


■  Initial  work 

-  Automatic  code  generation  process  created  to  support  rapid  algorithm 
development 

•  Identified  limitations  and  pitfalls 

•  Standardized  deployment  for  incorporation  in  object  oriented  simulations 

-  Original  Processes  developed  using  release  Matlab®  R1 1 

■  Ongoing  efforts 

-  Process  has  been  implemented  on  many  programs 

•  Hardware  models 

•  Control  algorithms 

•  Medium  and  high  fidelity 

-  Presently  using  Matlab®  Release  2009a 

•  Processes  updated  for  current  releases 


MDE  Processes  are  in  place  and  are  being  used  at 

Raytheon  Missile  Systems. 
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Intentions  of  Using  MDE  for  Integrated 
Flight  Simulation  Development 


Raytheon 

Missile  Systems 


■  MDE  is  a  powerful  process  for  designing  models,  both 
hardware  and  software,  for  simulations 

-  Because  of  requirements  imposed  on  IFSs,  impractical  to  develop 
entire  simulation  with  MDE 

■  Early  development  of  IFSs  requires  frequent  changes  to 
models 

-  Automatic  code  generation  from  MDE  methods  saves  time,  not  only  in 
initial  integration  of  the  model  into  the  IFS,  but  subsequent  changes  can 
be  made  simpler  and  quicker. 

■  While  much  initial  model  design  work  done  with  Simulink®, 
other  MDE  tools  are  used  to  develop  flight  software 


MDE,  when  used  appropriately,  is  a  powerful  tool  for  IFS  development 
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MDE  Tool  History  Example 


Raytheon 

Missile  Systems 


■  MDE  Tools  Evolve  Over  Time,  and  so  must  MDE  processes 

-  Matlab®  R2008a  and  Previous 

•  Would  generate  only  C  code 

•  C++  option  only  changed  the  file  extensions  from  “.c”  to  “.cpp” 

•  Early  versions  (R1 1 )  could  only  support  discrete  models 

-  Releases  Since  Matlab®  R2008b 

•  Includes  option  to  generate  “Encapsulated  C++”  code 

•  True  C++  class  that  can  be  instantiated  in  the  IFS  (multiple  times  if  needed) 

•  Includes  Initialize,  Step,  and  Finalize  member  functions 

•  Additional  member  functions  for  setting  or  getting  static  input  variables 


Continuous  MDE  tool  improvements  require  process  improvements 
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Model  Lifecycle  Comparison 


Raytheon 

Missile  Systems 
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MDE  Process  Flow 


Raytheon 

Missile  Systems 


Straightforward  process  using  MDE  models  to  develop 

Functional  Simulations 
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Feedback  changes  for  following  iterations 


Savings  Comparison 


Raytheon 

Missile  Systems 


■  Hardware  model  coded 

-  Control  Actuation  System  model 

-  Representative  model  for  most  hardware  models  integrated  in  IFS 

■  Used  three  methods  to  obtain  time  comparisons 

-  Hand-coded  from  Simulink®  block  diagram 
-Auto-coded  using  original  process  using  Matlab®  R11 

•  Can  only  use  discrete  blocks  and  integration  when  auto-coding 
-Auto-coded  using  updated  process  using  Matlab®  R2008b 

•  Continuous  blocks  and  integration  supported 

■  Note  that  process  times  are  for  a  first  pass  through  the  auto¬ 
coding  process 

-  Subsequent  integrations  of  the  same  model  should  show  even  further 
process  time  reductions 
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Savings  Comparison 


Raytheon 

Missile  Systems 


Task 

Hand-coding 

(hr) 

Auto-code  without 
Continuous  Block 
Support  (hr) 

Auto-code  with 
Continuous  Block 
Support  (hr) 

Create  usable  source  code  from  using  MDE 

Insert  and  connect  generic  I/O  port  content 

2 

2 

Replace  Integrators  with  ports 

2 

Continuous  block  identification  and  replacement 

8 

Auto-code  option  selection  and  code  generation 

<1 

<1 

Preparation  of  generated  code 

4 

4 

Handcoding  model  -  Simulation 

60 

Handcoding  model  -  Algorithm  Design  Tools 

60 

Subtotal 

120 

17 

7 

Common  efforts  to  integrate  code  into  IFS 

Modifying  IFS  wrapper  object,  input  files,  etc 

4 

4 

4 

Performing  unit  Tests  for  verification 

4 

4 

4 

Performing  Simulation  Tests  for  verification 

10 

10 

10 

Subtotal 

18 

18 

18 

Total  Conversion  Time 

138 

35 

25 

%  of  Hand-coding 

100% 

25.4% 

18.1% 

Significant  time  savings  when  auto-coding  models 


Performance  in  Integrated  Flight 
Simulations 


Raytheon 

Missile  Systems 


■  Currently  using  MDE  processes  in  simulations  on  multiple 

programs 

■  Extensive  verification  of  models  performed 

-  Developed  detailed  processes  for  conversion  of  the  model  to  C/C++ 
code 

-Verified  performance  of  the  models  integrated  in  the  IFS  match  the 
performance  of  the  original  model  as  a  unit  test 

-  Regression  runs  of  the  full  simulation  completed  to  verify  performance 
of  the  model  in  the  IFS 

■  Processes  updated  and  tested  with  latest  tool  capabilities 


Methodical  and  Thorough  Process  Used  in  Development 

of  IFSs  using  MDE  Methods 
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Performance  in  Integrated  Flight 
Simulations 


Raytheon 

Missile  Systems 


Wing  Actuation 
System  Hardware 
Model 


Piston  Displacement  (Simulation  vs  Simulink) 


Angle  Sweep  (Simulation  vs  Simulink) 


950 

900 

850 

800 

750 

700 

650 

600 

550 

500 


0.05  0.1  0.15  0.2  0.25  0.3  0.35 

s 

Tank  Temperature  [Simulation  vs  Simulink) 


Good  Agreement  in  Time  Domain  Performance 
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Performance  in  Integrated  Flight 
Simulations 


Raytheon 

Missile  Systems 


Good  Agreement  in  both  Time  and  Frequency  Domains 
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Common  Pitfalls 


Raytheon 

Missile  Systems 


■  Model  Configuration 

-  Every  model  is  different,  new  configurations  produce  new  problems 

-  Common  model  design  standards  needed  for  developers  to  streamline  integration  into  the 
simulation 

■  Tool  Capabilities 

-  As  with  any  tool,  user  must  understand  process,  model,  and  MDE  tool,  not  a  “push-button” 
process 

-  Common  areas  to  watch 

•  Timing  -  no  time  shift  present 

•  Does  auto-code  accurately  represent  the  system?  Auto-code  should  identically 
reproduce  outputs  given  identical  inputs 

■  Integration  Schemes 

-  Internal 

•  Continuous  -  Only  available  in  later  releases  of  Matlab® 

•  Discrete  -  Not  always  the  choice  of  model  developers  for  representing  system 

-  External 

•  Tie  into  simulation  numerical  integration  schemes 

•  Reduces  ability  to  verify  against  original  model 

While  MDE  tools  are  useful,  care  must  be  taken  in  model  development 
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Conclusions 


Raytheon 

Missile  Systems 


■  Raytheon  Missile  Systems  has  successfully  used  MDE 
processes  to  incorporate  models  into  IFSs 

■  Full  set  of  procedures  developed  to  aid  personnel  cross¬ 
program  and  to  train  new  users 

■  Procedures  verified  with  multiple  models  on  multiple 
simulations 

■  Procedures  are  updated  as  new  features  become  available  in 
MDE  tools 

■  Generating  code  automatically  using  MDE  processes  can 
save  significant  amounts  of  time  preparing  models  for 
incorporation  in  simulations,  and  can  be  completed  with 
confidence 
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Assurance 


System  assurance 

•  The  justified  confidence  that  a  system  functions  as  intended  and  is  free  of 
exploitable  vulnerabilities,  either  intentionally  or  unintentionally  designed  or 
inserted  as  part  of  the  system  at  any  time  during  the  life  cycle* 

Software  assurance 

•  Software’s  contribution  to  system  and  system  of  systems  (SoS)  assurance 

-  Software  assurance  in  the  context  of  a  system’s  and  SoS  mission  and  use 


Justified  confidence :  rational  basis  for  deciding  about  SoS  readiness  for  use 
Functions  as  intended :  involves  user  expectations,  which  change  over  time 
Environment  of  use 

•  Actual  environment  of  use  (not  just  the  expected  environment  of  use) 

•  Means  evaluating  robustness  against  unexpected  use,  threats,  and  changes 
the  environment 


Engineering  for  System  Assurance,  NDIA  System  Assurance  Committee,  2008,  www.acq.osd.mil/sse/pg/guidance.html 


Software  Engineering  Institute  Carnegie  Mellon 


Engineering  Improvement  in  Software 
Assurance:  A  Landscape  Framework 
October  2009 
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Problem  Scope 


Numerous  assurance  solutions  (i.e.,  technologies,  policies,  and 
practices)  are  available 

•  A  large  number  of  organizations  produce,  fund,  or  use  these  assurance 
solutions 

•  How  these  assurance  solutions  contribute  to  operational  assurance  is  often 
unclear 

Operational  environments  are  plagued  with  undiscovered  defects  and 
escalating  numbers  of  known  vulnerabilities 

•  Where  should  resources  be  invested  to  gain  the  most  benefit? 

•  Where  are  the  critical  gaps  in  available  assurance  solutions? 

•  What  additional  assurance  solutions  are  needed? 

•  Are  the  incentives  for  routinely  applying  assurance  solutions  effective? 


Software  Engineering  Institute  Carnegie  Mellon 


Engineering  Improvement  in  Software 
Assurance:  A  Landscape  Framework 
October  2009 

©2009  Carnegie  Mellon  University 


A  Solution  Approach 


Goal  -  longer-term 

•  Identify  gaps,  barriers,  and  incentives  to  the  formation,  adoption,  and 
application  of  assurance  solutions  (i.e.,  technologies,  policies,  practices)  to 
improve  operational  assurance 

•  Exploit  this  knowledge  to  accelerate  the  formation,  adoption,  and  application 
of  appropriate  assurance  solutions 

Near-term  approach 

•  Build  a  modeling  framework  that 

-  Characterizes  the  current  portfolio  of  organizations  working  in  assurance, 
available  assurance  solutions,  and  how  they  work  together  to  improve 
operational  assurance 

-  Characterizes  the  gaps,  barriers,  and  incentives  related  to  the  adoption 
and  application  in  operational  environments  of  assurance  solutions 

•  Leverage  (or  adapt)  existing  modeling  and  analysis  methods 


Software  Engineering  Institute  Carnegie  Mellon 


Engineering  Improvement  in  Software 
Assurance:  A  Landscape  Framework 
October  2009 
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Where  might  we  start? 


Key  Information  for  a  Modeling  Framework  to  Address 

1 

How  is  software  assurance  value  defined  for  a  selected  context? 

2 

Who/what  are  the  participating  organizations  and  assurance  solutions? 

3 

What  are  the  elements  of  value  exchanged  among  participants? 

4 

How  do  participating  organizations  and  assurance  solutions  work  together  to  achieve 
operational  assurance? 

5 

What  are  the  drivers  and  motivations  of  participating  organizations? 

6 

What  are  the  critical  usage  scenarios  and  behaviors  among  the  participating  organizations 
and  assurance  solutions? 

7 

What  are  the  adoption  and  operational  usage  mechanisms  used  for  assurance  solutions? 

8 

How  are  the  adoption  and  operational  usage  mechanisms  aligned  with  organizational 
context  and  need? 

9 

What  is  the  impact  of  future  trends  and  events  on  participating  organizations  and  assurance 
solutions? 

10 

What  patterns  of  possible  inefficiencies  can  be  identified? 

11 

What  are  candidates  for  improvements?  What  could  be  the  impact,  if  implemented? 

Software  Engineering  Institute  Carnegie  Mellon 


Engineering  Improvement  in  Software 
Assurance:  A  Landscape  Framework 
October  2009 
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Conceptual  Context  of  Assurance  Modeling 


includes  decision  makers, 
technologies,  practices,  people, 
and  their  relationships 


Engineering  Improvement  in  Software 
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Structure  of  Assurance  Modeling  Framework 


View 


Our  modeling  framework  is  comprised  of  multiple  categories  of  activities 
necessary  to  produce  an  assurance  capability  area  profile 

Each  activity  category  focuses  on  developing  insights  on  one  or  more  of  the 
framework  information  questions  and  produces  one  or  more  views 

Each  view  is  formed  using  one  or  more  methods 

A  profile  is  a  set  of  views  that  collectively  describe  an  assurance  landscape 
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Pilot  Use  of  the  Assurance  Modeling  Framework 


selected  assurance 
capability  area  fo£ 
analysis 


assurance  capabilities  drawr 
from  software  ecosystem  to 
support  assurance  properties 


Z 


Assurance 

Modeling 

Framework 


vulnerability 
management  selected 
as  the  Assurance 
Capability  Area 


assurance 

ecosystem 


Assurance 
Capability  Area 
Profile 


security  as  the 
assurance  property 
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Determine  Context 
and  Scope 


Characterize  Current 
State:  Participants 
Relationships 


Assurance 

Modeling 

Framework 


Characterize  Current 
State:  Asset  Maturation 
and  Adoption 


Determine  Future 
Factors 


Identify  Candidate 
Improvements 


Software  Engineering  Institul 


Critical  Context 
Analysis 


Value  Mapping 


SoS  Focus 
Analysis 


Driver  Identification 
&  Analysis 


System 

Dynamics 


Technology 
Development  & 
Transition^nal^sis^ 


Strategic 

Alternatives 

^nal^sis 


Principal 
Participants  & 
Influences 

Value 

Exchanged 

Potential 

Assurance 

Results 


Motivations 


Critical 

Behaviors 


Adoption  of 
Products 


Future  Drivers 


^  Inefficiencies 


Assurance 
Capability  Area 
Profile 


Prioritized 

Improvements 
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View:  Value  Exchanged  <02,3, 4) 


Method:  Value  Mapping 


•  Shows  static  relationships  among  principal  participants  (organizations 
and  assurance  solutions) 

•  Shows  primary  elements  of  value  exchanged  between  two  participants 


Selected  insights 

•  One  organization  or  technology  by  itself  does  not  mean  a  great  deal;  its 
relationship  to  other  organizations  and  technologies  has  meaning 

-  An  organization  may  play  several  roles  in  the  assurance  ecosystem 

•  Values  identified  in  value  exchanges  may  have  only  an  indirect  effect  on 
operational  assurance  and  is  often  difficult  to  determine 

•  The  models  provide  an  effective  way  for  assurance  solution  owners  to  describe 
and  better  understand  the  key  relationships  associated  with  their  solution 


Software  Engineering  Institute  Carnegie  Mellon 


Engineering  Improvement  in  Software 
Assurance:  A  Landscape  Framework 
October  2009 

©2009  Carnegie  Mellon  University 


12 


Legend 


^_k  Engineering  Improvement  in  Software 
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Sample  CVE  Value  Map  -2 


CVE  Diagram 


An  organization 
may  play 
multiple  roles 
with  different 
values 
exchanged 


Microsoft.  Apple. 
Symantec,  others 


CVE  -  Common  Vulnerability  Enumeration 
NVD  -  National  Vulnerability  Database 


Trusted  is  defined  as  being  CVE  knowledgeable  and  conforming  to  CVE  guidelines 
*  Public  source  vulnerability  data  Includes  advisories  and  alerts  from 
Reporting  Groups  and  Reporting  Product  Vendors. _ 
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View:  Potential  Assurance  Results  <Q2,4) 


Method:  SoS  Focus  Analysis 

_  _ I 


•  Produces  a  model  for  alignment  of  services  between  suppliers  of 
assurance  solutions  to  what  operational  users  do  to  achieve  operational 
assurance 

•  Oriented  to  defining  collaborations  within  complex,  socio-technical 
systems  (of  systems)  domains 


Selected  insights 

•  The  effect  an  assurance  solution  has  on  achieving  operational  assurance  is 
often  not  direct 

-  It  is  a  network  of  relationships  among  organizations  and  assurance  solutions 
that  must  be  understood  within  their  operational  context 

•  The  models  surface  potential  areas  of  inefficiencies  for  further  analysis 
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SoS  Focus  Analysis 
with  CVE 


What 

Vendors 

How 

CVE,  NVD 

1 

1 

i  i 

Addressing 

known 

vulnerabilities 

1 

Disseminating 
vulnerabilities 
and  patches 

1 

Maintaining 
current 
knowledge  of 
vulnerabilities 
and  patches 

i  Building, 

:  testing, 

issuing 
patches 

Registering 

Monitoring 

Supply-Side 

l 

2 

3 

Potential  inefficiencies: 

-  where  tacit  knowledge  is  held 
-  where  people  manually  synthesize  significant 
information  from  multiple  sources 


Who  I 

^^Who 

Whv 

Security  ^ 

Computer 

User 

analysts 

installations  & 

environments 

operations 

1 

1 

1 

1  1 

Maintaining 

1  \ 

Maintaining 

i  i 

Operational 

current 

awareness  of 

assurance  in 

knowledge  of 

risks  and 

the  context  of 

available 

effectiveness  of 

use 

patches  &  site 
configurations; 

solutions 

forming  site 

solutions 

Tracking, 

Installing 

Operational 

analyzing, 

solutions, 

availability  and 

forming 

monitoring 

integrity 

solutions 

effectiveness 

Demand-Side 

4 

5 

6 
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View:  Critical  Behaviors  (Q6) 


Method:  System  Dynamics 

I _ 1 1 _ I 

•  Produces  a  model  for  analyzing  critical  behaviors  within  complex 
socio-technical  system  of  system  domains 

•  Identifies  primary  positive  and  negative  feedback  loops  driving 
critical  behaviors 


Selected  insights 

•  There  is  a  tension  in  the  vendor  community  between  resources  for  proactive 
software  vulnerability  prevention  practices  and  reactive  patch  generation  and 
release  practices 

-  Urgency  of  response  has  historically  promoted  reactive  practices 

-  CVE-induced  market  pressures  are  beginning  to  promote  proactive  practices 

•  The  models  provide  a  structured  way  to  approach  discussions  among 
technology  representatives  and  other  affected  stakeholders 
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Sample  System  Dynamics  Model 


3.  The  proactive  approach  focuses  on  a  strategy  of 
vulnerability  prevention  based  on  applying  CWE 
information  within  the  vendor  community  to 
developed  software  that  prevents  vulnerabilities. 


1 .  Vendors  must  decide  how  to  split  resources 
between  reactive  and  proactive  responses  to 
product  vulnerabilities  to  balance  the  need  for  an 
immediate  response  with  the  need  for  a  proactive 
solution  that  prevents  product  vulnerabilities. 


endor  Community 
correcting  vul 
revention  problems 


Vendor  Community 
vuls  in  newly 
developed  software 


Proactive  Product 
Vulnerability 
Prevention 


Vendor  Community 
product  vuls 


Vendor  Community 
patching  product  vuls 


disseminating 
CWE  software 
weaknesses 


Vendor  Community 
vul  prevention  training 
and  experimentation 


Vendor 
Community 
resources  to 
vul  prevention- 


Vendor  Resource 
Reallocation 


Reactive  Product 
Vulnerability 
Patching 


Vendor 
Community 
resources 
to  patch  „ 


disseminating 
CVE  software 
vuls 


Vendor 

Communi 

urgency 

response 
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4.  If  vendors  feel  the  need  to  devote  more  resources 
to  vulnerability  patching  and  less  to  vulnerability 
prevention,  then  this  leads  to  a  downward  spiral  of 
increasingly  vulnerable  products  and  ever  increasing 
assurance  problems. 


2.  The  reactive  approach  patches  product 
vulnerabilities  based  on  CVE  information.  The 
development  of  patches  is  prioritized  based, 
in  part,  on  the  impact  a  given  vulnerability  is 
having  on  the  operational  community. 
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Summary 


Assurance  modeling  framework  lays  important  groundwork  by  providing 
a  multi-dimensional  approach  to 

•  Better  understand  relationships  between  organizations  and  assurance 
solutions  and  how  these  relationships  contribute  to  operational  assurance 

•  Begin  identifying  potential  areas  of  inefficiencies  across  a  spectrum  of 
technical  and  organizational  areas 

Status  of  SoS  software  assurance  modeling  framework  project 

•  Completed  initial  version  of  the  assurance  modeling  framework  and  validated 
it  through  the  pilot  on  vulnerability  management  as  a  selected  assurance 
capability  area 

•  Finishing  up  a  report  on  the  modeling  framework  and  its  pilot  use 
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Next  Steps 


>  Expand  modeling  of  future  trends  and  technology  formation  and 
adoption 

>  Review  the  behavioral  system  dynamics  models  with  community 
representatives 

>  Review  usage  scenarios  of  the  pilot  profile  with  community 
representatives 

>  Expand  the  use  of  the  framework  to  another  aspect  of  software 
assurance 
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•  Functional  analysis  gap  assessment 

—  Who  we  are 

—  What’s  the  situation 

—  What  we  did 

—  What  we  found 

*  Perspectives  that  might  apply  to  your  organization 

—  Perspectives  on  functional  analysis 

—  What  are  the  impacts  of  Model  Driven  Engineering  (MDE) 

—  How  are  things  evolving 

—  Some  common  recommendations  made  to  SSCI  members 
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About  the  Consortium 
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Systems  and  Software  Engineering  Practices 


Realizing  value  from  process  improvement 

•  Value-driven  process  improvement 

•  Quantifiable  business  performance  measures 

•  CMM®,  CMMI®  appraisals 


Life  cycle  strategies  for  complex  systems 

•  Disciplined  Agility 

•  Systematic  reuse  /  Product  lines 


Implementing  integrated  engineering 

•  Model-driven  engineering 

•  Requirements  analysis  &  automated  testing 

•  Architecture  and  design 

•  Security 

•  Measurement  &  analysis 

•  Verification  and  validation/Mission  assurance 


> 


j 


Applied  to  Member  Needs 

As  a  Consortium 

•  Co-funded  development 

•  Practitioner-led  training 

•  Technology  transfer 

As  a  Teammate 

•  Subject  matter  experts 

•  Process  consulting 

•  Technology  consulting 

As  an  Industry 

Association 

•  Voice  of  Industry 

•  Influence  govt,  agencies 

•  Best  practices/guidelines 

•  Neutral  ground/honest  broker 


Learn  more  at 

www.svstemsandsoftware.org 

with  a  For  Members  Only  account 
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•  Corporate  Overview 

—  5th  largest  U.S.  defense  contractor  and  a  leading 
manufacturer  of  business  jets 

—  2008  Sales  of  $29.3  billion 

—  Approximately  92,900  employees  operating  in:  United  States, 
Australia,  Austria,  Canada,  Germany,  Mexico,  Spain, 
Switzerland  and  the  United  Kingdom 

•  Business  Units  and  Operating  Groups 

—  Aerospace  -  1 5,300  Employees 

—  Combat  Systems  -  18,500  Employees 

—  Information  Systems  and  Technology  -  34,000  Employees 

—  Marine  Systems  -  22,000  Employees 
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Combat  Systems  Group 
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•  General  Dynamics  Land  Systems  (GDLS)  - 
Ground  and  Amphibious  Combat  Vehicles 

—  Headquartered  in  Sterling  Heights,  Michigan 

—  Operating  sites  in  Alabama,  California,  Florida,  Maryland, 
Michigan,  Ohio,  Pennsylvania,  Virginia,  Washington, 
Australia  and  Canada 

—  International  Programs  in  Afghanistan,  Australia,  Canada, 
Egypt,  Germany,  Iraq,  Israel,  Kuwait,  Morocco,  New 
Zealand,  Qatar,  Saudi  Arabia  and  the  United  Kingdom 

•  European  Land  Systems  -  Armored  Vehicles, 
Weapons  and  Ammunition 

—  Headquartered  in  Vienna,  Austria  with  operating  sites  in 
Germany,  Spain,  Switzerland 

•  Ordnance  and  Tactical  Systems  -  Ammunition 

—  Headquartered  in  St.  Petersburg,  Florida  with  operating  sites 
in  Alabama,  California,  Florida,  Illinois,  Pennsylvania,  Texas, 
Washington  and  Canada 

•  Armament  and  Technical  Products  -  Guns,  Detection 
Systems  and  Composites 

—  Headquartered  in  Charlotte,  North  Carolina  with  operating 
sites  in  Arkansas,  Maine,  Mississippi,  Nebraska,  Virginia 
and  Vermont 


Medium  Caliber  Ammunition 
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What’s  the  Situation? 
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GDLS’  processes  and  methods  for  functional  analysis 

have  the  following  objectives: 

1.  Perform  analysis  in  alignment  with  contract  and 
industry  standards 

2.  Reduce  cycle  time  through  continuous  improvement 

3.  Maintain  cycle  time  but  increase  value  by  greater 
through  put,  quality  and  consistency 

4.  Lead  design  by  maturing  interfaces,  functional 
requirements  and  performance  allocations  into  design 
cycle 
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What  We  Did  -  Approach 
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•  GDLS  requested  that  SSCI  perform  an  assessment  of 
their  functional  analysis  practices 

—  SSCI  worked  with  GDLS  team  to  identify  and  scope  the 
project  -  i.e.  include  anything  related  to  functional  analysis 
(e.g.,  processes,  skills,  technologies,  organizational 
dynamics,  etc.) 

—  Conducted  interviews/workshops  with  three  lines  of  business 
that  included  domain  experts,  SMEs  and  a  cross  section  of 
managers 

—  Captured  110  individual  items  of  feedback  that  SSCI  reduced 
into  about  20  general  categories 

—  Produced  technical  report  summarizing  the  detailed  findings 

—  Conducted  out-briefings  to  senior  management  on 
observations  and  recommendations 
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What  We  Found 
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•  Key  Findings  and  Insights: 

—  Good  processes,  practices,  and  training  have  been 
developed  and  are  in  use  by  GDLS  programs,  but  could  be 
more  tightly  integrated 

—  Modeling  and  simulation  is  being  used  to  provide  means  for 
assessing  performance  and  design  alternatives  to  support 
system  concept  selection 

—  Practices  are  evolving  to  leverage  model-driven  engineering 


*  SSCI  Recommendations 


—  Short-term  and  long-term  recommendations  addressing: 

-  Integrated  engineering 

-  Technology  adoption 

-  Customer  discussions  on  technology  and  process 
evolution 

-  Organization  and  responsibilities 

-  Leveraging  methods  and  domain  knowledge  expertise 
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Perspective  on  Functional  Analysis 


SYSTEMS  AND 
SOFTWARE 
CONSORTIUM 

BUILDING  BETTER 
SOLUTIONS  TOGETHER 


•  First  notable  methods  for  functional  analysis  is  often 
attributed  to  Lawrence  D.  Miles  who  was  a  design 
engineer  for  General  Electric  (1940s) 

—  Charles  Bytheway  extended  Mile's  functional  analysis 
concepts  and  introduced  the  methodology  called  Function 
Analysis  Systems  Technique  (FAST) 

—  Key  perspective  came  at  functional  analysis  from  a  value 
perspective 

—  Key  focus  is  on  alternative  designs  to  achieve  only  the 
required  functions  at  the  lowest  cost  while  meeting  the 
fundamental  requirements  of  the  customer 

•  As  system  complexity  increases  and  where  product 
lines  are  necessary,  a  broader  view  is  needed 

—  Functional  analysis  provides  inputs  to  architecting  which  are 
important  for  product  lines 
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Conceptual  Outputs  of  Systems  Engineering 
Process* 
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•  Fundamental 
information  from 
functional  analysis 
is: 

—  Derived 
requirements 

—  Performance 
requirements 

—  Interfaces 

•  Documentation 
of  alternatives 
and  decisions 


*Source:  Condensed  Guidelines  for  Successful  Acquisition  and  Management  of  Software-Intensive  Systems  Handbook 
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GDLS  Process  Extends  IEEE  1220  System 
Engineering  Process 
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Represents  the  interfaces  to  Functional  Analysis 


PROCESS  INPUTS 


r! 


1 


Requirements  Analysis 


Requirement  &  Constraint  Conflicts 


Requirements  Baseline 

Requirements  Validation 


I 


Requirement  Trade-offs  &  Impacts 


Validated  Requirements  Baseline 


Functional  Analysis 


Decomposition  &  Requirement 
Allocation  Alternatives 


I 


Functional  Architecture 


Functional  Verification 


Verified  Functional  Architecture 

Synthesis  J  _ 


Decomposition/Allocation 
Trade-offs  &  Impacts 


Design  Solution  Requirements 
&  Alternatives 


Design  Architecture 


Verification/Validation 


Design  Solution 
Trade-offs  &  Impacts 


Verified  Design  Architecture 


Systems  Analysis 

Requirements  Trade 
Studies  and 
Assessments 


Functional  Trade 
Studies  and 
Assessments 


Design  Trade 
Studies  and 
Assessments 


Systems  Integration  & 
Control 


T 


PROCESS  OUTPUTS 
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More  of  What  We  Found 
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•  GDLS  is  moving  towards  a  broader  view  of  functional 
analysis  through  the  integration  of: 

—  System  concept  selection  supported  by  computational-based 
decision  analysis  for  ranking  alternatives 

—  Modeling  &  simulation  supporting  performance,  dynamic, 
structural,  thermal,  functional  and  combat  analyses 

—  System-level  functional  analysis  that  traces  the  potentially 
significant  impact  on  lower-level  decisions  down  through 
subsystems 

—  Capture  operational  threads  that  cut  across  subsystems 

•  Integrating  functional  analysis  throughout  system 
engineering  and  subsystems  can  be  challenging 
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Integrated  Engineering 
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•  “Modern”  perspective  on  functional  analysis 

“Architect”  needs  to  factor  many  alternatives  into  how 
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Impacts  of  MDE  Adoption 
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•  Driver:  use  UML-based  notation  to  improve  communication 
with  software  engineers 

•  Barrier:  what  UML-based  notations  for  functional  analysis 
are  usable  by  system  engineers  and  understandable  by 
customers 

—  System  Modeling  Language  (SysML)  derived  from  UML  for  system 
engineers 

—  UML  and  SysML  are  designed  as  a  general-purpose  language 

—  Challenge  becomes  determining  what  modeling  artifacts  to  use  - 
how  and  when 

•  Benefits  of  automation  through  tailored  and  integrated 
tooling 

•  Technology  adoption  readiness  is  a  common  issue  with 
SSCI  members 
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Evolving  Systems  Engineering  through  Model  Driven  Functional  Analysis 


The  Four  Pillars  of  SysML 


SYSTEMS  AND 
SOFTWARE 
CONSORTIUM 

BUILDING  BETTER 
SOLUTIONS  TOGETHER 


1.  Structure 


req  VehideSpedfications 

]  R  =q  u  ir=  nre  rts  Cie^-am-  Braking  Req uirernents ’ 


Vehicle  System 
Specification 


^requirements 
Stopping  Distance 


id—  1  Dji, 

text-  Th  =  ve-hide  i  ^  all  =■  tc  c 
fr  ofTi  £  0  mph  ith  in  1  fO  ft 
cn  3  das p  dry  surface. 

- T - 


Braking  Subsystem 
Specification 


(■requirements 
A  nti  -L  oc  k  Perform  a  n  c  e 


id=  "337 ' 

t=xt=  Braking  subsystem  shall 
present  wheel  bd^up  uncer  all 
shaking  conditions.1 


■xderive  Re-sts 


ABS_Activ^ionSequence  [Sequence  Diagram] 


stm  i  re :  ra  ctic  n  [State  D  i  ag  ra  nijj 


2.  Behavior 

interaction 


act  P  re ventLa  cfcu  p  [Activity  D  i  agramlj 


X 


:DetectLossOf 
Traction 


'  ractLcss 


:  Mod  li  ate 
BrakingForae 


ra  ctLo  ss  v - 


- - w 


state 

machine 

activity/ 

function 


par [constraintBlock]  StraightLineVehicieDyTramics  [Parametric Diagram]^ 


tf  tl: 


tt 
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3.  Requirements 


QMG 

SYSTEMS 

MODELING 

LAH6UICE 


4.  Parametrics 
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SysML  Diagrams  Represent  System 
Engineering  Views 


SYSTEMS  AND 
SOFTWARE 
CONSORTIUM 

BUILDING  BETTER 
SOLUTIONS  TOGETHER 


•  SSCI  members  are  interested  in  how  SysML  supports 
key  system  engineering  processes  for  requirement, 
functional  analysis,  and  more 


SysML  Diagram 

z 

X 

Structure  Diagrams 

Behavior  Diagrams 

Use  Case  Diagram 


Activity  Diagram 


Sequence  Diagram 


Statechart  Diagram 


Requirement  Diagram 


IEEE  1220  SE  Process 


PROCESS  INPUTS 


i 


Requirements  Analysis 


Requirements  Baseline 


Requirements  Validation 

I  Validated  Requirements  Baseline 


Functional  Analysis 


I 


Functional  Architecture 


Functional  Verification 


Verified  Functional  Architecture 


Synthesis 


Design  Architecture 

Verification/Validation 

Verified  Design  Architecture 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  Log  No.  2009-107,  dated  10/05/09 

Copyright  ©  2009,  Systems  and  Software  Consortium,  Inc.  Version  1 .0  GENERAL  DYNAMICS 


16 


Evolving  Systems  Engineering  through  Model  Driven  Functional  Analysis 


GDLS  MDE-Related  Improvements 


SYSTEMS  AND 
SOFTWARE 
CONSORTIUM 

BUILDING  BETTER 
SOLUTIONS  TOGETHER 


•  Use  of  storyboards  addressing  need  for  better  CONOPS  that 
are  easy  to  understand  and  relevant  to  stakeholders  - 
tailored  activity  diagrams 

•  Generation  of  sequence  diagram  with  automation  to  move 
derived  requirements  to  be  linked  during  process  of 
elaborating  sequence  diagram 

—  Efficiency  and  completeness  step  to  ensure  derived  requirements 
moved  to  sequence  diagram,  where  traceability  links  are  added 

•  Allows  for  automated  movement  of  requirements  to  DOORS® 

•  Validation  through  execution 

•  Allows  for  automated  generation  of  measurement  data  for 
status/measurement  of  the  analysis  activities 
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Evolving  Systems  Engineering  through  Model  Driven  Functional  Analysis 


Technology  Adoption  Recommendation 


SYSTEMS  AND 
SOFTWARE 
CONSORTIUM 

BUILDING  BETTER 
SOLUTIONS  TOGETHER 


•  Don’t  “jump”  into  projects  without  determining  how  to  use 
MDE  tools 

—  One  SSCI  Member  said:  “after  training  we  know  how  the  tool 
works,  but  we  don’t  know  how  to  produce  work  with  the  tools” 

•  Use  pilot  projects  first  to  understand  the  “real”  value  of 
artifacts  produced  by  tools 

•  Don’t  over  estimate  value  of  results  in  comparison  to  effort 
needed  to  put  tools  and  processes  in  place 

•  Align  proposals  with  new  types  of  deliverables 

—  It  takes  more  time  to  produce  models  for  an  SRR  or  SFR  than  it 
does  using  a  document-centric  approach 

—  Documents  can  be  incomplete  (“good  enough”)  for  reviews,  but 
inconsistent  or  incomplete  models  cannot 

•  Prepare  for  tool  evolution  (version  upgrades) 
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Talk  with  Customer  About  Technology, 
Process,  and  Deliverable  Changes 


SYSTEMS  AND 
SOFTWARE 
CONSORTIUM 

BUILDING  BETTER 
SOLUTIONS  TOGETHER 


•  Typical  SRR  and  SFR  (PDR/CDR)  are  based  on  traditional 
document-centric  and  waterfall  lifecycle 

•  Model-based  approaches  result  in  artifacts  that  can  contribute  to 
multiple  reviews,  and  can  contribute  downstream  (e.g.,  V&V) 

•  Internal  and  external  stakeholders  need  to  understand  that  model- 
driven  upfront  work  is  superior  to  document-centric  analysis 

•  Successful  SSCI  member  example: 

—  Organization  adopting  new  modeling  practices  brought  customer  in  for 
multi-days  review  of  new  approach  and  deliverables 

—  Customer  recognized  increased  details  in  model-based  artifacts  that 
typically  don’t  exist  with  document-based  processes 

—  Customer  understood  how  details  would  be  beneficial  over  entire 
development  lifecycle 

—  Customer  was  pleased  and  wanted  to  know  why  other  projects  were  not 
using  modeling  approach 
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Points  to  Remember 


SYSTEMS  AND 
SOFTWARE 
CONSORTIUM 

BUILDING  BETTER 
SOLUTIONS  TOGETHER 


•  Integrated  engineering  is  challenging,  but  needed  as  impacts  of 
functional  analysis  decisions  can  cut  across  subsystems 

—  MDE  can  provide  many  efficiencies  and  help  link  functional 
analysis  information  so  that  impacts  can  be  identified  faster  and 
more  easily 

•  Technology  adoption 

—  Evolving  from  document-centric  to  model-centric  approach  requires 
coordination  with  stakeholders 

-  Models  can  take  longer  to  develop,  especially  the  first  time 

-  Models  identify  issues  early  and  provide  other  downstream  value 

—  Technology  readiness  can  impact  MDE  adoption: 

-  Know  what/how  models  map  to  current  processes 

-  Understand  tool  value  derived  from  operating  on  models 

-  Understand  how  tool  chains  can  support  the  entire  life  cycle 

-  Use  pilot  projects  to  prepare  for  adoption 

•  Consider  discussing  technology  and  process  evolution  changes 
with  customers/stakeholders 
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For  More  Information 


SYSTEMS  AND 
SOFTWARE 
CONSORTIUM 

BUILDING  BETTER 
SOLUTIONS  TOGETHER 


•  Questions  on  MDE 

—  Contact  Mark  R.  Blackburn,  Ph.D. 

—  561 .637.3452;  blackburn@svstemsandsoftware.org 

*  General  questions,  products,  services 

—  Main  Consortium  number  703.742.8877 
—  Home  page  www.svstemsandsoftware.org 
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Terms  and  Acronyms 


SYSTEMS  AND 
SOFTWARE 
CONSORTIUM 

BUILDING  BETTER 
SOLUTIONS  TOGETHER 


AADL 

Architecture  Analysis  &  Design  Language 

MMM 

Modeling  Maturity  Model 

AP233 

Application  Protocol  233 

MoDAF 

United  Kingdom  Ministry  of  Defence  Architectural 

BPML 

Business  Process  Modeling  Language 

Framework 

CAD 

Computer-Aided  Design 

MOF 

Meta  Object  Facility 

CASE 

Computer-Aided  Software  Engineering 

NASA 

National  Aeronautics  and  Space  Administration 

CATIA 

Computer  Aided  Three-dimensional  Interactive 

OCL 

Object  Constraint  Language 

Application 

OMG 

Object  Management  Group 

CDR 

Critical  Design  Review 

OO 

Object  oriented 

CMM® 

Capability  Maturity  Model 

PDR 

Preliminary  Design  Review 

CMMI® 

Capability  Maturity  Model  Integration 

PIM 

Platform  Independent  Model 

CONOPS 

Concept  of  Operations 

Pro/E 

Pro/ENGINEER 

CWM 

Common  Warehouse  Metamodel 

PSM 

Platform  Specific  Model 

DBMS 

Database  Management  System 

RFP 

Request  for  Proposal 

DoDAF 

Depart  of  Defense  Architectural  Framework 

ROI 

Return  On  Investment 

DSL 

Domain  Specific  Languages 

RTW 

Mathworks  Real  Time  Workshop 

GDLS 

General  Dynamics  Land  Systems 

SSCI 

Systems  and  Software  Consortium 

IBM 

International  Business  Machines 

Simulink/ 

Product  family  for  model-based  control 

ICD 

Interface  Control  Document 

Stateflow 

system  produced  by  The  Mathworks 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 

SCR 

Software  Cost  Reduction 

INCOSE 

International  Council  on  Systems  Engineering 

SDD 

Software  Design  Document 

IPR 

Integration  Problem  Report 

SE 

System  Engineering 

ISO 

International  Organization  for  Standardization 

SFR 

System  Functional  Review 

IT 

Information  Technology 

SME 

Subject  Matter  Expert 

Linux 

An  operating  system  created  by  Linus  Torvalds 

SQL 

Structured  Query  Language 

MAP 

Modeling  Adoption  Practices 

SRR 

System  Requirement  Review 

MARTE 

Modeling  and  Analysis  of  Real  Time  Embedded  systems 

SRS 

Software  Requirement  Specification 

MATRIXx 

Product  family  for  model-based  control  system  design 

SysML 

System  Modeling  Language 

produced  by  National  Instruments 

System  C 

IEEE  Standard  1666 

MBT 

Model  Based  Testing 

UML 

Unified  Modeling  Language 

MDA® 

Model  Driven  Architecture® 

XMI 

XML  Metadata  Interchange 

MDD™ 

Model  Driven  Development 

XML 

extensible  Markup  Language 

MDE 

Model  Driven  Engineering 

xUML 

Executable  UML 

MDSD 

Model  Driven  Software  Development 

Unix 

An  operating  system  with  trademark  held  by  Open  Group 

MDSE 

Model  Driven  Software  Engineering 

VHDL 

Verilog  Hardware  Description  Language 

MIC 

Model  Integrated  Computing 

VGS 

VxWorks 

T-VEC  Vector  Generation  System 

Operating  system  owned  by  WindRiver 
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T  rademarks 


SYSTEMS  AND 
SOFTWARE 
CONSORTIUM 

BUILDING  BETTER 
SOLUTIONS  TOGETHER 


•  OMG®,  MDA®,  UML®,  MOF®,  XMI®,  SysML™,  BPML™  are  registered  trademarks  or  trademarks  of  the  Object 
Management  Group. 

•  IBM™  is  a  trademark  of  the  IBM  Corporation 

•  DOORS  is  a  registered  trademark  of  IBM 

•  Java™  and  J2EE™  are  trademark  of  SUN  Microsystems 

•  XML™  is  a  trademark  of  W3C 

•  BridgePoint  is  a  registered  trademark  of  Mentor  Graphics. 

•  Java  is  trademarked  by  Sun  Microsystems,  Inc. 

•  Linux  is  a  registered  trademark  of  The  Linux  Mark  Institute. 

•  MagicDraw  is  a  trademark  of  No  Magic,  Inc. 

•  MATRIXx  is  a  registered  trademark  of  National  Instruments. 

•  MVS  is  a  trademark  of  IBM. 

•  Real-time  Studio  Professional  is  a  registered  trademark  of  ARTiSAN  Software  Tools,  Inc. 

•  Rhapsody  is  a  registered  trademark  of  Telelogic/IBM. 

•  Rose  XDE  is  a  registered  trademark  of  IBM. 

•  SCADE  is  copyrighted  to  Esterel  Technologies. 

•  Simulink  and  Stateflow  are  registered  trademarks  of  The  MathWorks. 

•  Statemate  is  a  registered  trademark  of  Telelogic/IBM. 

•  TAU/Developer  is  registered  to  Telelogic/IBM. 

•  T-VEC  is  a  registered  trademark  of  T-VEC  Technologies,  Inc. 

•  UNIX  is  a  registered  trademark  of  The  Open  Group. 

•  VAPS  is  registered  at  eNGENUITY  Technologies. 

•  VxWorks  is  a  registered  trademark  of  Wind  River  Systems,  Inc. 

•  VectorCAST  is  a  trademark  of  Vector  Software. 

•  Windows  is  a  registered  trademark  of  Microsoft  Corporation  in  the  United  States  and  other  countries. 

•  All  other  trademarks  belong  to  their  respective  organizations. 
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How  Do  AF  and  DoD  Define 
Systems  Engineering? 


“Air  Force  SE  involves  compreftens/Ve  planning ,  management ,  and 
execution  of  rigorous  technical  efforts  fo  develop,  field ,  &  sustain 
robust  products  and  systems.,. 

•  SE  collects^  coordinate s,  &  ensures  traceability  of  alj  sfafre/ro/der  needs  into 

aset  ofsystem  reguirements  through  a  balanced  process  that  takes  into 
account  effectiveness,  performance f  cost ,  schedule ,  and  risk.  ” 

AFI  63-1201,  Life  Cycle  Systems  Engineering 


•  Technical  Planning 

•  Requirements  IMgt 

*  Interface  Mgt 

*  Risk  Mgt 


*  Configuration  Mgt 

*  Technical  Data  Mgt 

*  Technical  Assessment 

*  Decision  Analysis 


Integrated  AT&L  Life  Cycle  Mgt  System,  V.5.3.4, 15  Jun  09 
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When  Does  AF  Say  SE  Should 
First  Be  Applied? 


“Application  of  SE  fundamentals  must  begin  with  concept  inception , 
and  must  cover  all  efforts  across  all  life  cycle  phases,  to  include 
sustainment  &  disposal,  for  ail  Air  Force  products  &  systems. 

Early  SE  provides  an  audit  trail  from  the  users’  capability  gaps  & 
needs,  through  concept  selection,  high-level  system  requirements 
refinement,  &  documentation  of  development  plans." 

AFI  63-1201,  Life  Cycle  Systems  Engineering 


AFRL/CC  will  ensure  incorporation  ofSE  methodologies  tailored  for 
AFRL  technology  development  done  in  support  of  evolutionary 
acquisition  programs. 

AFI  63-101 :  Acquisition  &  Sustainment  Life  Cycle  Management 
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Science  &  Technology  (S&T) 
Role  in  Early  SE 


AF  Early  Systems  Engineering  Guidebook  (vl,  Mar  09) 
states  the  following: 

A  technology  organization,  typically  AFRL,  works  with 

acquisition  organizations  to  ensure: 

•  Relevant  technologies  are  considered,  and  that  they  are  compatible 
with  the  desired  time  frame  and  expressed  acceptable  risk  levels 

•  New  approaches  made  possible  by  emerging  technologies,  as  well  as 
technologies  that  will  improve  a  system’s  effectiveness  and/or  reduce 
its  cost,  are  suggested 

•  Risks  and  uncertainties  associated  with  new  technologies  are 
estimated,  and  impacts  are  assessed 

•  Insight  as  to  user/operator  needs  is  gained,  allowing  technologists  to 
better  focus  their  technology  roadmaps 
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Does  Early  SE  Pay  Off? 


2006  Defense  Acquisition  Performance  Assessment  (DAPA) 
Project  Report  Survey  states: 

•  96%  of  respondents  cited  at  least  one  of  the  following  three  areas 
as  critical  to  maintaining  program  cost,  schedule,  and  performance 
(shown  in  ranked  order): 

-  Requirements  instability 

-  Funding  instability 

-  Tech  maturity 


•  The  greatest  trade  space,  and  thus  the  largest  risk  reduction 
opportunity,  exists  between  Milestones  (MS)  A  and  B 
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SE  Provides  a  Technical 
Foundation  for  Acquisition 


/lvis\ 


Business  Agreement 
Decisions  to  pursue 
a  material 
solution 


Engineering 
Support  ' 


Material  Solution  Analysis 


Technology  Development 


“Pre-Milestone 
A  and  Early- 
Phase 
Systems 
Engineering” 
Jan  2008 


Systems  Engineering  is  effective  when  it  informs, 
and  is  informed  by,  other  Acquisition  process  owners 
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AAR  Phase  II 
National  AAR  Team 
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AARII  Team  Combines  Warfighters  with 
Nationally  Recognized  Technologists 


—  Provides  Adverse  Weather  Operations 

,  .  Case  Number:  88ABW-2009- 

-  Improves  Fueling  Efficiency  4217,  Distribution,  Approved 

for  public  release; 
distribution  is  unlimited 


—  Improves  Pilot  Workload 


Significance  to  Air  Force 


*  Unmanned  Aerial  Vehicles 
—  Extends  Range 

—  Shortens  Response  for  Time-Critical  Targets 

—  Maintains  In-Theater  Presence  Using  Fewer  Assets 

—  Allows  Deployment  with  Manned  Fighters  and  Attack 
Without  the  Need  of  Forward  Staging  Areas 


Manned  Aircraft 


" How  does  it  (J-UCAS) 
air  refuel?  ...  which  is 
persistence  and 
endurance,  things  men 
can’t  do  in  airplanes  ” 

-Gen.  John  Jumper,  USAF, 
February  2005 


AAR  Assists  UCAVs  in  Reaching  Its  Full  Potential, 
and  Greatly  Enhances  Manned  Refueling 
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UCAS  Mission/AR  Overview 


Pre-Flight  Planning 


E n  Route 


Rendezvous 


Man  in  the  Loop  Sim 


Disconnect 
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Key  Technology  Challenges 


See  Near 

■  Determine  Relative  Position  with  Tankers 

Using  Position/Velocities  to  Close  Control 
Loop 

Hig  h  Confidence  in  Position  Accuracy 
Avoid  Aircraft  in  AAR  Area 

Collision  Avoidance 

■  AAR  Brings  Many  Aircraft  into  Same 
Airspace 


UC  AV  Must  Corttirtfursijsly  Determine 
Range  and  l  Lading  lo  Tanker  and 
Tauber  Velocities  from  20  Miles  to 
Station  keeping  Position 


Note;  UC  A  Vs  are  Flying 
“Formal  Lun  I  .ead" 


Command  and  Control 

■  Assure  UCAS  Accurately  Responds 

Boomer  Break-Away  Commands  Aircraft  Integration 


■  Minimize  impacts  to  tanker  fleet 

■  Fit  within  constrained  volume  of  UCAS 

■  Precision  control  of  UCAS 

■  Flight  critical  integration 

Real  World  Considerations 

■  Encryption,  latency,  drop-outs 

Case  Number:  88ABW-2009-4217,  Distribution: 

Approved  for  public  release;  distribution  is  unlimited  15 


AAR  Spiral  Approach  to 
Technology  Development 


Spiral  0  -  FY08 

■  TechnologyBase  Development 


Initial  Specifications 
ICDs  and  Architecture 
Research  PGPS  Prototype 
Fighter  CONOPs 
Sensor  Augmented  System 
Design/Requirements 


Spiral  1  -FY10 

■  PGPS  Adv  Prototype  Developme 

PGPS  Specifications 
PGPS  Prototype  System 


BomberCONOPs 


Spiral  2  -  FY12  t 

■  Sensor  Augmented  System  Tech  Matu  reffion 


-  AAR  System  Requirement  case  Number:  88ABW-2009-4217, 
AAR  Research  Prototype  Distribution:  Approved  for  public 
AAR  Arch  itecture/ICDs  release;  distribution  is  unlimited 


Precision  GPS  Closed-Loop 
Station  Keeping  Fit  Test  Aug  06 


■  Objectives: 

Evaluate  updated  Precision  GPS  (PGPS) 
performance 

Test  automated  formation  flight  in  contact 
position 

Evaluate  EO/IR  camera  as  AAR  sensor 

■  Accomplishments: 


Simplex  PGPS  relnav  and  automated  flight 
controls  held  Learjet  in  contact  position  for  23 
continuous  minutes 

Over  4  hrs  of  “hands-off”  formation  flight 

Over  85  minutes  of  “hands-off”  contact 
position  flight 

-  Found  sensor  suitable  for  AAR 


“The  System  Held  Contact  Position 
Better  than  I  Could”, 

Calspan  Test  Pilot 


Case  Number:  88ABW-2009-4217, 
Distribution:  Approved  for  public  release; 
distribution  is  unlimited 


AAR  UAS  Surrogate 


■  VISTA  Manned  Surrogate 

-  Autonomous  Capability  with 
Safety  Pilot  Override 

-  Variable  Stability  Flight  Controls 


■  Learjet  Manned  Surrogate 

-  Autonomous  Capability  with 
Safety  Pilot  Override 

-  Variable  Stability  Flight  Controls 


Case  Number:  88ABW-2009-4217,  Distribution: 
Approved  for  public  release;  distribution  is  unlimited 
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AAR  Station-Keeping  Flight  Test 
Example  Contact  Position  Performance 


■  FY10  Precision  RELNAV  Open-loop  Flight  Test  (PROFT)  (Learjet) 

Simultaneously  collect  GPS  data  from  multiple  LN-251  s  for  maturing  GPS 
Demonstrate  the  TTNT  Redundancy  Features  on  tanker 

■  FY10  RELNAV  Open-Loop  Flight  Test  (Learjet) 

Verify  fixes  from  PROFT 

Evaluate  EO/IR  camera  as  AAR  sensor 

■  VISTA  Inner-Loop  Flight  Test 

Characterize  RELNAV  Redundancy  Architecture 
Validate  A/C  model  in  VISTA 

■  VISTA  Station  Keeping  Flight  Test  (SKFT) 

Evaluate  Precision  GPS  (PGPS)  performance 
Evaluate  formation  flight  control  System 
Evaluate  EO/IR  camera  as  AAR  sensor 

■  VISTA  Positions  and  Pathways  Flight  Test(PPFT) 

Demonstrate  end-to-end  AAR  CONOPS  mission  including  wet  hookup  and  contingency 

■  Full  CONOPS  Simulation 

Demonstrate  full  CONOPS  with  multiple  tankers  and  receivers 
Demonstrate  mission  control  capability  from  AVO  to  receivers 


Case  Number:  88ABW-2009-4217,  Distribution: 
Approved  for  public  release;  distribution  is  unlimited 
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Outline 


•  Early  Systems  Engineering  (SE)  in  Acquisition 

•  Technology  Maturation  (Tech  Mat)  in  Early  SE 

•  AAR  Program  Background 

•  Tech  Mat  Planning  for  AAR 


Case  Number:  88ABW-2009-4217,  Distribution: 
Approved  for  public  release;  distribution  is  unlimited 
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Key  Technical  Objectives 
for  AAR  Phase  II 


Reduce  key  risks  to  transition 

•  Technical  maturity,  documentation,  and  impressions 


Mature  AAR  technology  into  a  robust  design,  with 


supporting  analysis  and  specifications 

•  Safety,  LO-compatibility,  system  health  monitoring, 


FMECA,  specifications 

•  Develop  prototype  system  based  on  design 
demonstrating  feasibility  of  design 
•  Robust  testing  of  prototype  to  determine  adequacy  of 


design  and  failure  modes 


Case  Number:  88ABW-2009-4217,  Distribution: 
Approved  for  public  release;  distribution  is  unlimited 
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Automated  Aerial  Refueling  (AAR) 
Phase  II  Way  Forward 


Demonstrate  AAR  in  a  relevant 
environment  through  wet  hookup 


Focus  areas  for  further  maturation 


Redundancy/contingency  management 

Multi-ship  operations 

Sensor  augmented  (GPS+EO/IR  Sensor) 
positioning  system 

Robust  AAR  System/CONOPS  Simulation 
Full  AR  CONOPS  flight  test  with  hookup 


Case  Number:  88ABW-2009-4217,  Distribution: 
Approved  for  public  release;  distribution  is  unlimited 
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Critical  Elements  of  AAR  II 
Technology  Maturation  Planning 


•  Technology  Participants 

•  Technology  Demonstration  Plan 

*  Requirements  Documentation 

•  Program  Objective 

*  Approach 

*  Technology  Development  Required 

*  Applicable  Systems 

*  Product/Payoff/Exit  Criteria 

•  Programs/ Activities  Related  to  AAR  Success 

•  Programs/Missions  Supported  by  AAR 

•  Technology  Milestones  List 

•  Technology  Deliverables 

*  Risk  Analysis 

•  Technology  Protection  Plan 


Case  Number:  88ABW-2009-4217,  Distribution: 
Approved  for  public  release;  distribution  is  unlimited 
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Critical  Elements  of  AAR  II 
Technology  Maturation  Planning 


'Acquisition  Strategy 

*  Target  Acquisition  Programs 

*  Stakeholders 

*  Capability  Document 

*  Availability  Dates  for  Technologies 

*  Functional  Strategies 

•  Flight  Qualification 

•  Airworthiness  Certification 

*  Environmental  Qualification 

*  Logistics  Support 

•  Technology/Acquisition  Bridge 


Case  Number:  88ABW-2009-4217,  Distribution: 
Approved  for  public  release;  distribution  is  unlimited 
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Elements  of  Management  Plan  to 
Mature  Requirements  for  AAR 


•  Process  Flow 

•  Change  Requests 

•  Requirements  Change  Control  Board 

•  Tools 

•  Products 

*  Baseline 

*  Traceability 

*  Verification  Methods 


Case  Number:  88ABW-2009-4217,  Distribution: 
Approved  for  public  release;  distribution  is  unlimited 
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CONSTELLATION  TECHNOLOGY 


Analyzing  Your  World 


Upgrade  Fluid  System  Filter  Element  Monitoring 
to  Increase  Operational  Reliability  and  Support 
Condition  Based  Maintenance  Capability 


Presented  by 
Gary  Rosenberg 
October  29,  2009 


LATioN technology  pjiter  Monitoring  Supports  CBM 

ilyzing  Your  World  I  I 


Why  the  filter  element? 


Filters  are  already  incorporated  in  all 
important  systems  to  provide  operational 
reliability. 


Fluid  systems  such  as;  Transmission, 
Lubrication,  Hydraulic,  Fuel  and  Electronic 
Cooling  can  utilize  this  effective  CBM  process. 
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In-System  Filter  Element  Monitoring  can  support  the 
following  major  levels  of  Condition  Based  Maintenance: 

1 .  Identify  when  filter  element  service  is  required. 

2.  Determine  the  remaining  filter  element  service  life, 
the  asset’s  mission  availability  and  establish  a 
schedule  for  filter  element  service. 

3 .  Provide  an  early  indication  of  a  system  fault. 


Copyright  ©  2 009  by  Constellation  Technology  Corporation,  All  Rights  Reserved. 
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Basic  Transmission  Lubrication  System 

Rotor  Output 


Turbine  Input 


Heat  Exchanger 
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Exchanger 

System  Simplification 


System  Components 

Valves,  Bearings,  Gears,  Pump 

\ 

System  Filter 

Lubrication  Fluid  Flow 
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Fluid  Cleanliness 
(Particles  /  Fluid  Volume) 


constellation technolo^^^  Filter  Monitoring  Supports  CBM 

Analyzing  Your  World  I  I 


Lubrication  Fluid  Flow 
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Fluid  Cleanliness 
(Particles  /  Fluid  Volume) 


constellation technolo^^^  Filter  Monitoring  Supports  CBM 

Analyzing  Your  World  I  I 


Lubrication  Fluid  Flow 
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Fluid  Cleanliness 

(Particles  /  Fluid  Volume) 


constellation technolo^^^  Filter  Monitoring  Supports  CBM 

Analyzing  Your  World  I  I 
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Single  Pass  of 
Fluid  Through 
The  System 


C/3 

C/3 

<D 


o 

E 


Fluid  Volume  Passing  through  the  System 


Filter 

Loading 
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Single  Pass  of 
Fluid  Through 
The  System 


C/3 

C/3 

<D 


o 

E 


Fluid  Volume  Passing  through  the  System 


Filter 

Loading 
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Single  Pass  of 
Fluid  Through 
The  System 


C/3 

C/3 

<D 


o 

E 


Fluid  Volume  Passing  through  the  System 


Filter 

Loading 
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Single  Pass  of 
Fluid  Through 
The  System 


C/3 

C/3 

<D 


o 

E 


Fluid  Volume  Passing  through  the  System 


Filter 

Loading 
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Single  Pass  of 
Fluid  Through 
The  System 


C/3 

C/3 

<D 


o 

E 


Fluid  Volume  Passing  through  the  System 


Filter 

Loading 
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Single  Pass  of 
Fluid  Through 
The  System 


CO 

GO 

<D 

1 

O 

•  i—H 

E 


Abnormal 
Equilibrium 


Fluid  Volume  Passing  through  the  System 


Normal 

Filter 

Loading 
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Single  Pass  of 
Fluid  Through 
The  System 


CO 

GO 

<D 

1 

O 

•  i—H 

E 


Abnormal 
Equilibrium 


Fluid  Volume  Passing  through  the  System 


Abnormal 
Filter  Loading 
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Single  Pass  of 
Fluid  Through 
The  System 


CO 

GO 

<D 

1 

O 

•  i—H 

E 


Abnormal 
Equilibrium 


Fluid  Volume  Passing  through  the  System 


Abnormal 
Filter  Loading 
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Single  Pass  of 
Fluid  Through 
The  System 


CO 

GO 

<D 

1 

O 

•  i—H 

E 


Abnormal 
Equilibrium 


Abnormal 
Filter  Loading 


Fluid  Volume  Passing  through  the  System 


Sum  of  Filter 
Element  Loadings 
Over  Time 


Oh 

O 

U 

Q 

cd 

$-h 

2 

c/5 

CO 

CD 

$-h 

PLh 

$-h 

<D 


Abnormal 
Loading  Curve 


Terminal 
Pressure  Drop 


Deviation  from  normal  loading  curve 
provides  an  Early  Indication  of  System  Fault 
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Single  Pass  of 
Fluid  Through 
The  System 


CO 

GO 

<D 

1 

O 

•  i—H 

E 


Abnormal 
Equilibrium 


Abnormal 
Filter  Loading 


Fluid  Volume  Passing  through  the  System 


Sum  of  Filter 
Element  Loadings 
Over  Time 


Oh 

O 
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cd 

$-h 
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c/5 

CO 

CD 

$-h 

PLh 

$-h 

<D 


Abnormal 
Loading  Curve 


Terminal 
Pressure  Drop 


CBM  Level  3 

Deviation  from  normal  loading  curve 
provides  an  Early  Indication  of  System  Fault 
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Comparison  of  Differential  Pressure  Monitors 


Monitor  Capability 

Indicator 

Switch 

Indicator/Switch 

Sensor 

Port  Mounting  Compatible 

yes 

yes 

yes 

yes 

Validate  Operation 

no 

yes1 

no 

yes 

Continuous  output 

no 

no 

no 

yes 

Remaining  Life 

no 

no 

no 

yes 

Mission  Availability 

no 

no 

no 

yes 

Schedule  Service 

no 

no 

no 

yes 

Service  Filter  Element 

yes2 

yes3 

yes2 

yes 

Early  Fault  Indication 

no 

no 

no 

yes 

Missing  Filter  Element 

no 

no 

no 

yes 

System  In  Bypass 

no 

no 

no 

yes 

1  Only  on  systems  having  high  pressure  drop  during  cold  start  without  thermal  lockout 

2  No  validation,  Poor  operational  reliability,  Possibly  system  fault 

3  Possible  validation,  Poor  operational  reliability,  Possibly  system  fault 
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Upgrading  to  a  Differential  Pressure  Sensor  to  provide 
real-time  in-system  monitoring  of  the  filter  element’s 
performance  can  support  CBM  in  addition  to  providing: 


•  Improved  indication  tolerance 

•  No  moving  parts,  robust  design 

•  An  integrated  temperature  sensor  output 

•  Full  utilization  of  the  filter  element 

•  Reduction  of  required  filter  changes 

•  Improved  reliability  and  operational  readiness 
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Challenges  and  Benefits  of  Applying 

ISO  STEP 


Stuart  T.  Booth 


Charlie  Stirk 


Systems  Engineering  Directorate 

Office  of  the  Director,  Defense 
Research  and  Engineering 


PDES  Inc. 

Technical  Advisory  Committee 
Chairman 


12th  Annual  NDIA  Systems  Engineering  Conference 

October  29, 2009 


NDIA  SE  Conference:  Challenges  and  Benefits  of  Applying  ISO  STEP 
10/29/2009  Page-1 


UNCLASSIFIED 


Outline 


•  Overview  of  the  Enterprise  Challenge 

•  Tool  Vendor  and  Data  Management  Challenges 

•  OSD  Path  forward  on  a  Common  Data  Exchange  Approach 

•  Overview  of  APs 

-  AP-233  and  AP-239 

-  Status  of  AP-233 

•  AP  Architecture 

•  AP  Methodology  for  implementation 

•  Summary 
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Notional  Enterprise  View 
“Synchronizing  the  Layers” 


Data 
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Tool  Vendor  and  Data  Management 

Challenges 


•  Tool  vendors  are  in  competition  resulting  in  no  incentive  to 
integrate  tool  capabilities... creating  “islands  of  automation” 

-  Proprietary  Data  Interfaces 

-  Proprietary  Data  Model 

•  The  tool  customer  must  invest  in  tool  integration  to 
overcome  the  tool  interface  challenges 

-  Point  to  Point  tool  interfaces. .  .custom  scripts 

•  Typical  SE  tools  consist  of: 

-  Requirements  Management 

-  Architecture 

-  Integration  and  Test 

-  Risk  Management 

•  All  stakeholders  suffer  due  to  poor  data  integration  and 
synchronization  and  consistency  problems. 


Need  another  approach.... 

Common  Data  exchange  protocol  that  facilitates  data  integration. 
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Notional  Multi-Tool  Vendor 
Typical  Consequences 


•  Engineers  must  query  multiple  databases  to  get  a  full  picture;  painful  process  with  temporary  value 

•  Maintaining  synchronization  and  consistency  is  nearly  impossible 

•  Expert  tool  staff  is  needed  to  develop  custom  scripts  to  support  point  to  point  data  exchanges 
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Notional  Single  Tool  Vendor 
Typical  Consequences 


•  Tools  have  proprietary  interface  and  not  easily  integrated  with  other  tools 
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•  Supports  multi-tool  environment.... One  size  does  not  fit  all 

•  Data  Synchronization  and  consistency  is  achieved. 

•  Non  proprietary  data  exchange  interface  facilitating  ease  of  sharing  data 

•  Engineers/Users  can  query  and  report  data 
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DoD  Approach  Going  Forward 


•  Investigating  Common  Data  Exchange  Approach 

-  ISO  STEP  AP-233  (For  Systems  Engineering) 

•  This  has  its  challenges 

-  Developing  a  robust  data  model  to  support  systems  engineering 

-  Applying  AP-233  in  a  commercial  environment  (flexibility  and  ease  of  implementation) 

-  Getting  the  vendors  to  work  with  you 

•  OSD  Systems  Engineering  is  funding  a  study  through  the  Systems 
Engineering  Research  Council  (SERC)  to  investigate  the  maturity 
of  AP-233.  This  study  will: 

-  Evaluate  the  current  AP-233  (systems  engineering)  and  AP-239  (for  logistics)  data 
models  for  compatibility. 

-  Implement  a  working  prototype  of  AP-233  with  market  leading  systems  engineering 
tools. 

-  Adapt  AP-233  to  selected  specialty  systems  engineering  tools. 

-  Adapt  AP-233  to  capture,  maintain  and  export  cost  &  schedule  (EVM)  data 

-  Assess  the  ease  of  use  and  competencies  needed  to  apply  AP-233  for  commercial 
application 

-  Make  recommendations  on  a  path  forward. 
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Industrial  Data  Standards 


International  Organization  for  Standardization 
(ISO)  Subcommittee  TC184/SC4 

-  Describe  and  manage  industrial  data  through  product  life 

-  2007  ISO  Eicher  Leadership  Award 

-  For  reusable  information  model  building  blocks  to  enforce 
interoperability  and  simplify  implementation 

(Past  winners  include  MPEG  &  ISO  9000) 

Areas  from  product  design,  analysis  & 
manufacture 

-  STEP  -  Standard  for  the  exchange  of  product  data 

-  MANDATE  -  Industrial  manufacturing  management  data 

-  PSL  -  Process  specification  language 

-  PLIB  -  Parts  library 

-  Process  Plants  including  Oil  and  Gas  facilities  lifecycle  data 

-  eOTD  -  electronic  open  technical  dictionary  for  catalogs 
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STEP  for  DoD  Acquisition  Cycle 


A 


<> 


Materiel 

Solution 

Analysis 


Technology 

Development 
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(Program  Initiation) 


Technological  Maturity 
and  Integration  Risk 
Assessment 

Competitive 
Prototyping 


(pdr)0 


A 


Engineering  and  Manufacturing 
Development 


0 


Production  and  Deployment 


0 


FRP 

Decision 

Review 


Operations  and 
Support 


AP  233 -Systems  Engineering  Data  Representation 
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Mandates  for  ISO  10303  STEP 


•  requirement  for  Computer-Aided  Engineering,  Design  and  Manufacturing  systems 
used  by  NASA  to  have  interchange  tools  that  support  ISO  10303 

-  NASA-STD-2817  Chief  Information  Officer  (1999) 

•  “procure  all  product/technical  data  in  attachment  (1)  digital  formats  and  ensure 
product  model  data  meets  ISO/STEP  requirements  specified  in  attachment  (1).” 

-  ASN  RDA  Memo  by  John  Young,  Oct.  23,  2004,  STEP  for  2-D  and  3-D  CAD 

•  “implement  a  similar  approach  that  adopts  ISO  10303  to  enhance  interoperability” 
as  described  in  Young  memo  above 

-  OSD  ATL  Memo  by  Ken  Krieg,  June  21 , 2005,  STEP  for  UID 

•  “Ratifying  nations  agree  to  apply  ISO  10303-239  for  product  data  management  in 
cooperative  NATO  acquisition  programs.” 

-  NATO  STANAG  4661 ,  ratified  by  US 

•  “The  PM  shall  require  the  use  of  International  Standards  Organization  (ISO)  10303, 
Standard  for  Exchange  of  Product  (STEP)  Model  Data,  AP239,  Product  Life  Cycle 
Support,  for  engineering  data  “ 

-  AFI  63-101  ,  April  17,  2009,  PLCS  for  engineering  data 
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Modular  STEP  AP’s 


•  Application  Protocol  (AP)  Modularization 
Benefits 

-  Faster  revision  process  (months  rather  than  years) 

-  Interoperability  of  implementations  thru  reuse 

•  Modular  AP  Domains 

-  AP203  Mechanical  CAD  (parts  &  assemblies) 

-  AP209  CAE  (FEA  and  CFD) 

-  AP210  EDA  (aka  ECAD,  components  to  racks) 

-  AP233  Systems  Engineering 

-  AP239  Product  Life  Cycle  Support  (PLCS) 


NDIA  SE  Conference:  Challenges  and  Benefits  of  Applying  ISO  STEP 
10/29/2009  Page-12 


UNCLASSIFIED 


AP233  is  One  Enabler  of 
INCOSE  2020  Vision 


Model  based  systems  engineering 

(MBSE) 


Past 


•  Specifications 

•  Interface  requirements 

•  System  design 

•  Analysis  &  Trade-off 

•  Test  plans 


Future 


-  MBSE  is  the  formalized  application  of  modeling  to  support 
system  requirements,  design,  analysis,  verification  and  validation 
beginning  in  the  conceptual  design  phase,  and  continuing 
throughout  development  and  later  life  cycle  phases. 


INCOSE 
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SE  and  PLCS 


*  PLCS  shares  over  80%  of  models  with  AP233  SE 

*  Implementations  of  PLCS 

-  Pilots  at  NAVAIR,  NAVSEA  &  Electric  Boat 

-  Production  use  at  US  Army  TARDEC  &  contractors 

-  US  Army  LOGSA  developing  PLCS  DEX’s 

-  Extensive  use  by  NATO  allies  for  ships,  aircraft,  land  vehicles 

*  PLCS  Edition  2  in  ISO  standardization  process 

-  Fixes  issues  found  in  implementations 

-  Incorporated  into  233  through  common  modules 

-  Incorporates  new  capabilities  from  AP233  SE 

-  Risk,  Decision  Support,  V&V 
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tgi  What  Does  AP233  Enable? 


*  Program  management 

-  Issue 

-  Activities 

-  Approvals 

-  Risk 

-  Probability  & 
Consequence 

-  Source  &  Impact 

-  Contingency  plans 

-  Project 

-  Organizational  structure 

-  Project  breakdown 

-  Schedule 

-  Work  structure 

-  Management  information 
resources 


*  System  modeling 

-  Decision  support 

•  Requirements 
management 

•  Measures  of 
effectiveness 

•  Analysis  interface 

•  Verification  &  Analysis 

•  Justification 

-  System  structure 

•  Product  data 
management 

•  Breakdown 

•  Interface 

-  System  behavior 

•  Function  based 
behavior 

•  State  based  behavior 
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SE/PLCS  Overlap 
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AP233  Implementations 


•  Migration  between  versions  of  SE  tools 

-  UGS  Slate  to  UGS  Systems  Engineering 

•  Exchange  between  Requirements  Management  tools 

-  IBM  Requisite  Pro  and  Telelogic  DOORS 

•  Model  management  of  SysML  and  interoperability  with 
other  domains 

-  i.e.  Risk,  Program/Project,  downstream  CAD/CAM,  PLCS 

•  DoDAF  to  AP233  for  exchange  and  archive 

-  CADM  representation  of  DoDAF  views 

•  Multi-domain  simulation  management 

-  Requirements  through  analysis  -  EU  Vivace  &  MBEST 

•  Earned  Value  Management  XML  Schema  mapping  into  233 
reference  data 

-  Associate  cost  &  schedule  with  systems  engineering 


NDIA  SE  Conference:  Challenges  and  Benefits  of  Applying  ISO  STEP 
10/29/2009  Page-17 


UNCLASSIFIED 


Scripting  API  Implemented 


•  Application 
Programming  Interface 
(API)  in  simple  and 
accessible  language 

•  Programmer  must 
know  concepts, 
attributes  and 
relationships  in  AP233 

•  Ruby  API  code 
generated  from  AP233 
EXPRESS  ARM 

•  Available  as  open 
source  from 
www.exff.org 
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Proposed  High  Level  API 


•  Efficient  Access: 
classes  group 
objects  that  are 
create  or  destroyed 
simultaneously 

•  Business  Objects:  at 
level  of  SE  domain 
concepts  for 
mapping  to  software 
tools 

•  Web  Services: 
functions  of  SE 
domain  separate 
from  data  structures 
for  integration 


SE/AF 

Service 

Oriented 

Applications 


Choreographed 
Business 
Processes 


Web 

Services 


Svcl 


Operational 
Systems  & 
Persistent 
Databases 


AP233  can  define 
envelope  content 
of  Web  Service 


AP233  APIs  can 
implement  Web  service 


SE/AF  Web  Service 


AP233  API  Library 


AR233-RDB  Binding 
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Extending  233/239  with  Domain 

Semantics 


DEX 


Scope 


fCJittCJit  tCXtfCXt 

text 

Busirsesscontext 

Taxttext  ,-axt 
is  Jit  text  text  text 
text. 

Description 

Tftdiext  Jrtri 
text  text. 

!tW 

tort  test  ' 

Definition  ‘ 
template 
template 

Reference  data 
class 
class 


provides  guidance 
for  usage 
..of  ISO  10303-239 


provides  guidance” 
fa*  usage  of 


is  composed  a 


Template 

Definition 

PLCSentity 

PLCSentity 

Parameters 


,.dallnGd  In - 

a.  #  *■  - 


Reference  data 
class 
class 


reference  Id 


OASIS  RD 

Class  (term+deFinilion) 
Gass  (temn+deriniuon} 

□ass  (tfirrr.+deflmfcon) 
□ass  (term+definitKin) 

r  Class  (term+deFinilion) 


reference  to 


specifies  \. 
usage  of 


Capability 

Scope 

ex?  tett  text  text 

test. 

Ekisinesseontext 

reuse? i  text  text 
text 

Description 

template 

Text  text  text 
text  text 

template 

Text  text  text 
text  text 


usage  guidance  for 


specializes 


ISO  10303  239  PLCS 

(the  formal  standard) 


•  Reference  data  in  Web  Ontology  Language  (OWL)  tailors  to  domain 

*  Templates  are  assembled  into  Data  Exchange  Specification  (DEX) 
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Reference  Data  Issues 


*  Need  expert  knowledge  of  STEP  information 
models  to  properly  subtype  with  reference  data 


*  Many  potential  sources  of  reference  data  from 
different  domains  (need  domain  experts  involved) 


*  Basic  set  theory  used  to  classify  reference  data 

*  Potential  for  other  uses  of  OWL  (e.g.,  semantic 
web,  reasoning) 
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DoDAF  and  AP233 


•  There  exists  a  CADM-AP233  OWL 
representation  (www.exff.org) 

-  Used  AP233  WD2  version  with  fixes,  CADM  1 .02 

-  Need  to  update  to  current  version  of  AP233  and 
newer  versions  of  CADM  (1.5) 

-  For  legacy  program  analysis  and  data  migration 

•  Need  to  map  UPDM  with  AP233 

-  SysML  portions  from  NIST  FutureSTEP  project 

-  Need  to  map  UPDM  extensions  to  SysML 
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SysML  Issues 


•  XMI  -  XML  Metadata  Interchange 

-  For  UML  and  others  expressed  in  OMG  Meta 
Object  Facility  (MOF) 

-  Vendor  implementations  currently  incompatible 

-  OMG  Model  Interchange  Working  Group  to 
improve  interoperability 

-  Expect  will  be  solved  and  XMI  will  serve  for  most 
inter-SysML  tool  exchange 

•  Model  configuration  and  other 
management  aspects  out  of  scope  for 
SysML,  but  provided  by  AP233 
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SysML  and  AP233 


•  SysML  to  AP233  mapping  underway 

-  Both  based  on  INCOSE  concept  model 

-  Creating  reference  data  for  SysML 

-  Consensus  is  SysML  info  a  subset/subtype  of 
AP233 

•  233  enhances  SysML  by 

-  Management  and  representation  of 

-  Risk,  Analysis,  Fine-Grained  Configuration,  Program/Project ... 

-  Linking  to  downstream  CAD,  CAE,  CAM,  PLCS 

•  EXPRESS  meta-model  now  in  MOF 

-  233  first  test  case  of  bringing  STEP  AP  into  OMG 
MDA 
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Earned  Value  Management 


Government  contract  cost  and  schedule  performance 
reporting 

Standards  for  EVM  Systems 

-  ANSI/EIA-748-A  EVMS  Guidelines 

-  XML  Schema  based  on  ANSI  X.12  806  &  839 


-  NDIA  Program  Management  Systems  Committee  XML  Working  Group 

-  Defense  Contract  Management  Agenc 

EVM  Central  Repository 

-  Required  for  major  programs 

-  Used  for  analysis 

-  PM  software  vendors  support 

Mapping  EVM  into  233 

-  OWL  Reference  Data 


Need  to  map  EVM  Schema  and  233-based  Schema 

-  Maintains  upward  compatibility 
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233  Pulls  it  All  Together 


•  STEP  AP233  can  integrate  cost,  schedule  and 
systems  engineering 

*  Models  can  be  managed,  inter-related,  and  linked 
to  specialty  engineering  domains 


Project  Management 
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AP233  is  designed  to 


*  Capture  system  requirements,  design  and 
analysis  data  over  the  life  cycle 

*  Support  interoperability  and  integration  of 
Systems  Engineering  tools 

*  Provide  a  “front  end”  to  PLCS-based  Support 
Engineering  tools 

*  Link  with  detailed  design,  PDM,  analysis,  etc. 
tools  through  other  STEP  protocols 

*  Align  with  OMG  SysML  and  UPDM  for  DoDAF 

*  Enable  INCOSE’s  vision  of  Model-Based  Systems 
Engineering 
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For  Additional  Information 


Stuart  Booth 
stuart.booth.ctr@osd.mil 

Charles  Stirk 
stirk@costvision.com 
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Purdue 

UNIVERSITY 


School  of  Aeronautics  and  Astronautics 


Dynamic  Modeling  of  Programmatic  and 
Systematic  interdependence  for  System  of 

Systems  Acquisition 


Daniel  DeLaurentis 

Email:  ddelaure@purdue.edu 


Brian  Sauser 

Email:  bsauser@stevens.edu 


Muharrem  Mane 

Email:  mane@purdue.edu 


Alex  Gorod 

Email:  aaorodis@stevens.edu 
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Overview  of  Agenda/Presentation 

•  Motivation  and  problem  statement 

•  Recap  from  prior  work 

-  Conceptual  model  based  on  OSD’s  SoS  SE  Guide 

-  Computer  simulation:  Exploratory  SoS  Acquisition  Model 

•  Snapshots  from  illustrative  problems 

-  Dynamic  impacts  of  risk 

-  Implementation  of  system-specific  risk 

-  Impact  of  system-specific  risk  and  SoS  network  topology 

•  Summary 


Purdue 

UNIVERSITY 


Sjnrrrr 


School  of  Aeronautics  and  Astronautics 


Motivation 

Literature  on  recent  history  indicates  a  variety  of  challenges  for  SoS  acquisition 


S5**irr 


Purdue 

UNIVERSITY 


School  of  Aeronautics  and  Astronautics 


SoS  Sources  of  Complexity 


Working  Definition  for 

Complexity: 

the  amount  of  information 
necessary  to  describe  the 
regularities  in  a  system 
effectively 


•  Dynamic  and  Uncertain  Connectivity 


•  between  levels  of  abstraction 
•across  scope  dimensions 

•  “Porous”  boundary 

•Changes  in  constitution  of  SoS 

•  Heterogeneity  &  Multiplicity 


multiple  time  scales 

emergence  (unforeseen  interdependencies) 
Evolving  nature  of  an  ‘open  system’ 


•  Multiplicity  of  perspectives:  A  root  cause  of  interoperability  issues 


•Heterogeneity  of  participants  (within  and  between  Human  &  Technical); 
Socio-Technical  Systems 


Purdue 

UNIVERSITY 


E5*rrrr 


School  of  Aeronautics  and  Astronautics 


Root  Causes  of  Failure 

(within  acquisition  processes) 

•  Misalignment  of  objectives  among  the  systems 

•  Limited  span  of  control  of  the  SoS  engineer  on  the 
component  systems  of  the  SoS 

•  Evolution  of  the  SoS 

•  Inflexibility  of  the  component  system  designs 

•  Emergent  behavior  revealing  hidden  dependencies 
within  systems 

•  Perceived  complexity  of  systems 

•  Challenges  in  system  representation 

Used  categories  from  Rouse,  W.  (2007,  June).  Complex  Engineered,  Organizational  and  Natural  Systems.  Systems  Engineering,  10,  3.,  pp.  260-271 
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Purdue 

UNIVERSITY 
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Recap:  Research  Goals 

•  Uncover  underlying  functions  affected  by  complexities  due  to  evolution 
in  SoS  acquisition  and  span-of-control 

•  Capture  Dynamics:  Exploratory  SoS  Acquisition  Model 

-  Depicts  the  processes  (SoS  SE  Guide)  in  a  hierarchical  setting 

-  Show  the  flow  of  control  between  the  processes  throughout  the  acquisition 
life-cycle 

-  Interactive  computational  model:  allow  users  to  ‘explore’  complexities 

•  Experiment:  Generate  insights  and  approaches  to  improve  the 
probability  of  program  success 


•  Mapping  of  Operational  Views  (OV)  to  Systems  Views  (SV) 
-  System  capabilities  and  their  interconnections 
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UNIVERSITY 
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Recap;  Development  of  a  Dynamic, 
Exploratory  Model  for  SoS  Acquisition 

1.  Pre-Acquisition  Model  (not  included  here) 

-  Understand  the  influence  of  external  stakeholders  on  the 
acquisition  process 

2.  Acquisition  Strategy  Model 

-  Based  on  the  16  technical  management  and  technical  systems 
engineering  processes  outlined  in  the  Defense  Acquisition 
Guidebook  (5000  series)  applied  to  an  SoS  environment  (SoS- 
SE  Guide) 

-  Conceptual  model  depicts  the  processes  in  a  hierarchical  setting 
to  show  the  flow  of  control  between  the  processes  throughout 
the  acquisition  life-cycle 


Purdue 

UNIVERSITY 


School  of  Aeronautics  and  Astronautics 


Recap:  Acquisition  /  Development  -  The  Paper  Model 

(based  on  SoS  SE  Guide) 

Project-level  (SoS) 

Risk  profile:  low,  med,  high 
Span-of-control:  low,  high 


Risk  Level 


Span-of-Control 


Acqusition  Strategy 

Core  processes  as  identified  in  the 
DoD  SoS-SE  Guidebook 


t3(0) 


t4(0) 


Inability  to  provide 
feasible  design 
solutions  results  in 
changing  the 
requirements  for  the 
next  iteration 


Solution 

i 

Tech  Planning 

Tech 

Assessment 

- - - - - 

Optimal 

Design 

Solution 


t6(0) 


t5(0) 


Technical 

Requirements 


'  Validation  and  Verification  provides 
feedback  regarding  conflicting  inter¬ 
system  requirements,  impractical 
design  solutions  and  speedy 
recognition  potential  emergent 
behavior.  Thus,  acting  as  emergent 
behavior  detector 


Validation 

Verification 

— 

Requirement,  Data,  Risk, 
Configuration,  Interface 
Management 


|  I  Implement  sys  A  |  ; 

|  Integrate  sys  A  | 

!  | Implement  sys  B  |  | _ 

Integrate  sys  B  | 

|  Implement  sys  N  |  I 

|  Integrate  sys  N  | 

Requirement-level 

•  Number  of  requirements 

•  Requirement  dependency 

•  Probability  of  disruption 

System-level 

•  System  dependency 

•  Initial  completeness  level 

•  Int/lmp  time 

•  Probability  of  disruption 
(comes  from  risk-profile) 


Output 
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Methodology  Abstraction 


^ .  Operational  capability  (derived  from  SoS) 


\  Requirements  /  Activities 
\  (OV-2,  OV-5) 

/ .  Systems  /  Programs 

(SV-1 ,  OV-2) 


y 

/ 


V 

i 


->  - 


\  —  i 

_ _  j-" 


Operational  (OV):  systems  work 
together  to  provide  a  capability 

System  (S V):  define  nature  of 
interaction  between  systems 

Programmatic:  relationship 
between  systems  during 
development 


•  Discrete-event  simulation  with  probabilistic  behavior  of  systems 


Institute  ofTechnology 


Levels  have  predetermined  probability  of  disruption 

•  Requirement-level  disruptions:  affect  design  solutions  (i.e.  design  solution  of  system  X 
cannot  meet  requirement) 

•  System-level  disruptions:  affects  completeness  level  of  system  and  completion  time  (i.e.  set 
back  in  implementation  phase  of  system  X  results  in  longer  time) 
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Requirement  1 

- 1 

Requirement  2 

Sys  B* 

Illustrative  Example 

Z\  SoS 

\  Requirements 

. '  "  2  ''' 


System  Dep  (R1 )  System  Dep  (R2) 
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Effects  of  Disruptors 
(system-level) 

Inevitable  disruptions  on  both  system-level  and  requirement  levels  will  occur 
Technology  Assessment  is  able  to  immediately  trace  and  resolve  the  problem 
-  This  prevents  the  development  from  stalling  or  regressing  over  multiple  time-steps 

Requirement:  1  System:  a 


Each  color  represents  an 
individual  system  (system  ‘a9 
is  blue) 


Negative  disruptions  correspond  to 
system  re-engineering  and  lower 
completeness  level  in  Integration 
(and  Implementation)  phase 
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Effect  of  Project  Risk 


(determines  probability  of  disruption  in  Integration  and  Implementation  phase) 


Requirement:  1  System:  a 


Requirement:  1  System:  a 


80  100 


Low-risk  instance 


High-risk  instance 


•  Some  projects  have  a  much  higher  risk  factor 

-  They  are  more  vulnerable  to  negative  disruptions  in  their  development 

•  Higher  risk  of  disruptions  implies  more  time  to  complete  stages 


-  In  fact,  completion  may  fail  ->  return  to  Design  Solution 
•  Not  all  systems  in  a  SoS,  however  have  the  same  risk-level 
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Impact  of  System-Specific  Risk 

•  Quantify  the  impact  that  system-specific  risk  has  on  the 
completion  time  of  the  SoS 

-  Measure  risk  in  a  SoS  network 

-  Observe  changes  in  completion  time  due  to  different  risk-levels 


•  Example  problem 

-  One  requirement  and  three  component 
systems 

-  Each  system  can  have  a  distinct  risk-level 

-  Risk-level  indicates  probability  of 

disruption  in  implementation  &  integration 
phase 

-  Risk  for  the  SoS  varies  as  the  level  and 
combinations  of  system-specific  risk  change 

-  Wan  to  capture  the  effect  of  these  changes 
and  measure  the  risk  for  t  he  entire  SoS 


Requirement 


ems  &  Enterprises 


A 


B 


Network-Risk  Metric 


•  Consider  the  following  network-risk  metric/index  VS  c 


N  N 


i= 1  7=1 


where  is  the  risk  of  system  j  and  it  has  values  of  1 , 2,  or  3  (for  low,  mid, 
and  high  risk)  and  A  is  the  adjacency  matrix  (system  interdependencies) 

•  The  network-risk  metric  is  a  dimensionless  number  and  considers 
the  system-risk  and  the  system  dependencies  simultaneously 

•  Current  implementation  does  not  yet  consider  the  higher-order 
system  interdependencies  (cascading  effects  of  risk) 

•  i.e.  system  A  is  impacted  by  system  B,  but  system  B  is  also 
impacted  by  system  C;  risk  of  system  A  should  be  more  than 
just  the  sum  of  the  risk  of  system-A  and  system-B 
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Exploratory  Model  Experiments  a 

Experiment  set-up 

-  Each  system  can  have  a  low,  mid,  or  high  risk-level 

•  A  total  of  27  combinations  for  the  3-system  network 

-  Run  Monte  Carlo  simulation  of  Exploratory  Model  (500  samples) 
Experiment  results 

-  Capture  impact  of  system-specific  risk  on  SoS  completion  time 

-  Identify  critical  system  and  risk  combination 


♦ 


sys-C  risk=high 
sys-C  risk=mid 
sys-C  risk=low 


mid 

system-B  risk 


low  low 


mid 

system-A  risk 
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Impact  of  System-Risk  and  SoS  Network  Topology 

•  Previous  experiment  captured  the  impact  of  system  risk  for  a  fixed  SoS 
network 

•  It  is  also  possible  to  consider  the  impact  of  system-specific  risk  coupled  with 
different  network  topologies 

•  Consider  30  randomly  generated  SoS  configurations 


-  Uniformly  random  selection  of  number  of  systems  (up  tol  0  systems) 
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Exploratory  Model  Experiments 

•  Experiment  set-up 

-  For  each  system  in  each  SoS  network  randomly  generate  a  risk-level 

-  Run  Monte  Carlo  simulation  of  Exploratory  Model  (500  samples)  for 
each  SoS  network 


•  Experiment  results 

-  Capture  impact  of  system-specific  risk 
AND  network  topology  (i.e. 
interdependencies)  on  SoS  completion 
time 

•  Observations 

•  SoS  with  higher  risk  metric/index 
have  higher  completion  time 

•  Scatter  potentially  due  to  the  higher- 
order  impact  of  risk  (i.e.  cascading 
effects) 
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network  risk  metric,  R 
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Observations 

•  Exploratory  model  is  intended  to  enable  acquisition 
professionals  and  program  engineers  to  learn  about 
complexities,  dynamics,  and  disruptions,  identifying  markers  of 
failure  and  success 

-  Evolution  of  interdependencies 

-  Network  structure  and  span-of-control  of  SoS 

•  Current  implementation  if  system-risk  seems  to  capture  the 
right  things 

•  System-specific  risk  and  SoS  network  topology  experiments 
are  a  means  to  compare  different  SoS  options  that  may  satisfy 
the  same  requirement 

•  Shortcomings 

-  R  does  not  capture  the  higher  order  impact  of  dependencies 

-  Current  efforts  focused  on  addressing  this 
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Thank  You 
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Effect  of  Span-of-Control 
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Imp  Int 
Decision  Analysis 
Design  Solution 
Logical  Analysis 
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Design  Solution  - 
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High  Span-of-control 


Requirement:  1 
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Low  Span-of-control 


STEVENS 


•  Span-of-control  has  large  impact  on  project  time 

•  High  span-of-control  ->  SoS  level  authority,  can  implement  in  parallel 
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DoD  Directions  for 
Life-Cycle  Support 
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■  Reduce  operating  and  support  (O&S)  costs  for  deployed 
weapon  systems 

■  Minimize  the  logistics  footprint  for  deployed  weapon  systems 
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One  Way  to  Get  There 
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■  Design  into  our  weapon  systems  lower  value  Line 
Replaceable  Units  (LRUs) 

■  Line  Replaceable  Modules  (LRM) 
is  one  solution 

■  I  will  discuss  some  considerations/issues  of 
implementing  LRMs 
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LRM  vs.  LRU 
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Line  Replaceable  Module 


Line  Replaceable  Unit 


LRM  is  more  cost  effective,  light  weight 
and  easier  to  remove/replace 
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Environmental  Design 
Considerations 


Raytheon 


■  Vibration 

-  Shock  mounting  approach 

■  Temperature 

-  Operating  temperature  requirements 

-  Not-in-use  requirement  (non-mission  use) 

■  ESD 

-  Prevent  ESD  damage  due  to  any  environmental  factors 
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BIT  Design  Considerations 


Raytheon 


Fault  detection  vs.  fault  isolation 

-  Harder  to  isolate  to  LRM 


Fault  Isolation  is  easier  at 
the  LRU  level. 


< 


r 

REX  Unit 

Receiver 

Preprocessor 

Waveform  Gen 

Transmit  Drive 

Local  Oscillator 

Linear  Regulator 

V 

Chassis 

> 


Fault  Isolation  is  harder  at  the 
LRM  level 


BIT  Maturity:  higher  development  costs  — ►  lower  support  costs 
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Maintainability  Design 
Considerations 

■  Accessibility  for  maintenance 

-  Installation  trades 

•  New  platform 

•  Major  modification  to  existing  platform 

-  Access  to  module  for  testing  and  R/R 

•  Access  points  for  testing 

•  Space  to  open  covers/doors 

•  Room  to  attach  cabling  and  hoses 

■  Special  Tooling/Support  Equipment 
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Unique  Identification  (UID) 

■  Total  Asset  Visibility  (TAV) 

■  Pedigree  of  the  weapon  system 

■  Real-time  information 

■  History  of  the  item  (how  many  times  it  has  come  back, 
maintenance  history,  repair  history,  FRACAS) 

■  By  tracking  at  the  LRM  level,  faster  trends,  design 
implementation,  and  design  changes 

are  identified  (as  long  as  it  is  the  same  format/fit/function) 
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Storing 


Raytheon 


■  Storing 

-  Less  storing  space  required  for  LRMs  due  to  smaller  size 

-  Under  250  lbs  can  utilize  overnight  express  shipping 

•  Less  docking  space  required 

•  Less  time  to  transport 

•  May  reduce  the  need  of  special  storage  containers  (i.e.;  ISOPODS) 

-  Reduce  the  footprints  in  supply 
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Sparing 


Raytheon 


■  Spare  the  high  failure  LRMs  vs.  LRUs 

-  Quantity  based  on  weighted  failure  rates 


■  Spare  testing 
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MTBF 

REX  Unit 

16,300 

— K  Much  higher  quantity  to 
*  spare  at  LRU  level 

Receiver 

190,000 

Preprocessor 

79,000 

— ^  Higher  quantity  in  spares 

Waveform  Gen 

75,000 

y  for  these  2  LRMs 

Transmit  Drive 

88,000 

Local  Oscillator 

93,000 

Linear  Regulator 

202,000 

LRM  delivers  much  less 
redundant  supply  - 
Supply  more 
representative  of  the 

Chassis 

332,000 

actual  failure  items. 
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Summary 


Raytheon 


■  Therefore  LRMs  can  significantly  reduce  the  logistics 
footprint  due  to  fewer  spares  quantity  and  smaller  physical 
dimensions  of  each  spare. 

■  Reduced  O&S  cost  can  be  achieved  due  to: 

-  Decreased  MTTR  (Mean  Time  To  Repair) 

-  Requires  less  manpower  effort  to  accomplish  remove/replace 

-  Minimized  spare  investment  based  on  reliability  at  the  lower  level  (LRM) 
of  the  weapon  systems 

-  Less  special  support  equipment 

-  Reduce  shipping  cost  and  faster  shipping 
turnaround  time 
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Product  Life  Cycle 


LIFE  CYCLE 


Program  Management  Process 

Systems  Engineering  Processes 

Development 

Supply 

Production 

Maintenance 

Modifications  y' 

Depot 

Sustainment  View  of  DoD  5000.02 
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Real  View  of  SE  Process 


Incorrect  Assumptions 


*  Developers  work  with  a  blank  canvas  - 
sustainers  are  given  a  painted  picture 

-  Once  a  concept  is  created,  the  canvas  is  no  longer 
blank  for  anyone 

-  It  becomes  a  continuous  improvement/refinement 
process  from  that  point  on 

•  Sustainment  SE  processes  are  different  than 
acquisition  SE  processes 

-  Processes  and  process  objectives  are  the  same 

•  Process  implementation  can  vary 

•  Organizational  responsibility  can  vary 

•  Domain  knowledge  required  can  vary 

-  Every  cycle  through  the  SE  process  reassesses  prior 
decisions  for  cost,  schedule,  and  performance 
soundness 
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SE  Processes  per  DAG  Chapter  4 


Technical  Management 
Processes 

•  Decision  Analysis 

•  Technical  Planning 

•  Technical  Assessment 

•  Requirements  Management 

•  Risk  Management 

•  Configuration  Management 

•  Technical  Data  Management 

•  Interface  Management 


Technical  Processes 


•  Stakeholders  Requirements 
Definition 

•  Requirements  Analysis 

•  Architectural  Design 

•  Implementation 

•  Integration 

•  Verification 

•  Validation 

•  Transition 


Some  SE  Process  Examples 


•  Technical  Planning  (e.g.  SEP) 

-  Modifications 

-  Engineering  authority  /  MRBs 

-  CCB  procedures  /  membership 

-  OSS&E  characteristics 


-  Data  repository 

-  Master  documentation  updates 

-  Maintenance  data  systems 

-  etc 


•  Risk  (Opportunity)  Management 

-  New  technology  considerations 

-  Disposition  of  DMS  issues 

-  Resolution  of  aging  issues 

-  TOC  reductions 

-  etc 


SE  Processes  -  Plain  English 
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Technical  Processes 


•  Analyze  customer  needs 

•  Convert  customer  needs  into  system  level 
performance  requirements 

•  Allocate  and  derive  system  level  performance 
requirements  into  performance  requirements  for 
system  pieces 

•  Develop  design  solutions  for  performance 
requirements  of  system  pieces 

•  Verify  design  solution  meets  performance 
requirements 

•  Validate  design  solution  satisfies  customer  needs 


FOR  SVsy. 


Can’t  Buy  Parts  -  Now  What? 


Buy  enough  spares  to  last  through  the  product’s 
remaining  life 


•  Cannibalize  parts  from  other  systems  and  end-items  in 
the  inventory 

•  Acquire  technical  data/rights  and  qualify  (1)  a  new 

supplier  or  (2)  in-house  production  ^ 

/  SE 

•  Qualify  a  new  design  to  an  existing  performance  (  Sustainment 

specification  V  Tasks 

-  One  for  one  _ ^ 

-  Many  for  one  (e.g.  replace  3  existing  boxes  with  1  new  box) 

•  No  data  available  case 

-  Identify/measure  operational  environment 

-  Define  requirements  and  conduct  tests  to  compare  existing 
part  with  new  part 

-  Use  new  part  if  as  good  or  better  than  existing  one 


SE  Processes  -  Plain  English 

- -  4«raar 

Technical  Management 
Processes 


•  Make  technical  decisions 

•  Plan  the  technical  management  of  the  program 

•  Conduct  technical  studies 

•  Analyze  technical  information 

•  Document  decisions 

•  Develop  backup  approaches  for  risky  areas 

•  Manage  technical  changes 

•  Develop  and  maintain  technical  information 
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Technical  Baseline 


Customer  /  User 
Program  Direction 
Preferences 
Needs 


Configuration 


Program 
Performance  Reports 
Deficiency  Reports 
Aging  Trends 
Certification 


Production  / 


Specs  /  Dwgs  /  Code  List 
Product  Specific 
Data 


Maintenance 
Facilities 

Training  /  Certification 


Supply 
Vendors 
Spare  Parts 
DMS 


Technical  Baseline 


*  Definition  -  all  of  the  technical  information 
needed  to  support  a  product  throughout  its  life 
cycle 

*  Many  different  approval  processes  involved 

-  Configuration  change  control 

-  Maintenance  procedures 

-  Verification 

-  Validation 

-  Certification 

-  etc 

*  All  of  the  information  needs  to  be  archived 
and  maintained  throughout  a  product’s  life 
cycle 


Configuration  Information 


Configuration  Baselines 


Products  /  Processes 


<ssr 


Configuration  Baselines 
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Specifications 


•  Definition  -  contains  both  requirements  and 
verification  methods  in  one  “document” 

-  Requirement  documents  (SRDs/TRDs)  missing 
verification  methods 

•  Product  types 

-  System 

-  Item 

-  Software 

-  Process 

-  Material 

•  Other  types  include  -  Interface 

-  Don’t  buy  interfaces  --  buy  to  an  interface 


Configuration  Baseline  Control 
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Configuration  Control  Boards  (CCBs) 


-  Focus  on  configuration  baseline  documentation 

-  Engineering  change  proposals  (ECPs) 

-  Non  conformance  (waivers,  deviations,  variances,  etc) 

-  Can  be  used  to  establish  baselines 


*  ECP  Classification 

-  Class  I 

•  Change  form,  fit,  or  function 

•  Note:  Changing  the  length  of  a  decal  is  a  Class  I  change 

-  Class  II 

•  Everything  else  (minor  corrections) 


Defining  Class  I  as  gov’t  control  and  Class  II  as 
contractor  control  is  incorrect 


Configuration  Changes 


'  Identify  ' 

[  Configuration  l 

^  Change  / 

v.  ✓ 


The  PM  Is  Responsible  For  The  Data 


Configuration  Baseline  Control 


*  Material  Review  Boards  (MRBs) 

-  Used  to  disposition  minor  non  conformances 

-  Mirrors  Class  II  ECP  approval  /  delegation 

*  Critical  /  Major  Non  Conformances 

-  Requires  CCB  approval 

*  Supply  Prime  Vendor  Contracts 

-  May  allow  parts  substitution 

-  Changes  the  configuration  when  it  happens 


Product  Specific  Data 


*  Requirements  and  interface  management 
information  not  incorporated  into 
configuration  baseline  documentation 

*  Actual  product  configuration 

-  Product  built  against  a  specific  configuration 

•  Part  numbers  /  serial  numbers  /  lot  numbers  /  stock 
numbers  /  etc 

•  Maintenance  procedures  and  data 

•  Verification  /  validation  reports 

•  Etc 

*  Verification  information  /  tools 

-  Test  plans  /  procedures 

•  Demonstrated  performance  /  market  standards 

-  Number  of  test  articles  /  test  sequence 

-  Modeling  and  simulation  tools 

-  Analytical  tools 


'♦a 


^STit>  itf  of  ^ 


^<ear 


Verification  /  Validation 


Verification  /  Validation 


•  Verification:  Satisfies  configuration  baselines 

-  Developmental  test  and  evaluation 

-  Usually  performed  by  contractor  with  government 
observation 


•  Validation:  Satisfies  customer  /  operational  user 
needs  (i.e.  capabilities) 

-  Operational  test  and  evaluation 

-  Performed  by  customer  or  operational  user 


Verification 


Qualification 

X 

1 st  Article  /  Accelerated  Aging  / 
Surveillance 

X 

% 


^®F/Ti  itp  of  ^ 
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Product 

Acceptance 


Acceptance  / 
QA 


Verification 


Gov’t  Development: 


Final  Thoughts 


•  F3I  (Form,  Fit,  Function,  &  Interface)  Replacement 

-  New  component  must  be  verified 

-  Really  F2I:  form  changes 

*  Depot 

-  The  only  engineering  authority  the  depot  has  is  what  it’s 
given  in  the  Work  Specification 

-  Work  Specification  mandates  maintenance  procedures 
depot  is  to  use  to  make  repairs 

•  Phrase  -  Make  “X”  repair  using  best  commercial 
practice  -  gives  depot  authorization  to  use  their  own 
repair  procedures 

-  Same  Work  Specification  used  for  both  in-house  and 
contracted  depot  work 


Final  Thoughts 


•  Government  Supply  Chain  Managers 

-  Not  program  managers 

-  Responsible  for  stock,  store,  and  issue  tasks  only 

-  DLA  Directive  3200.1  states  services  retain  engineering 
and  configuration  management  responsibility  for  the  parts 
DLA  buys 

-  Applies  even  if  not  classified  as  a  critical  safety  item 

•  Can’t  find  old  performance  specifications  -  check 
out  bidder  packages  /  CDs 


%^Tuteo^° 


Developing  an  Introductory  Course  in: 


Model-Based  Systems  Engineering  (MBSE) 

with  the 

Systems  Modeling  Language  (SysML) 

Joe  Wolfrom 


October  29,  2009 


APPLIED  PHYSICS  LABORATORY 


Outline 


■  Purpose  of  the  Presentation 

■  MBSE/SysML  Course  Challenge 

■  Course  Background  Information 

■  Context 

■  Purpose 

■  Demographics 

■  Text  and  Software  Used 

■  Coverage 

■  Course  Schedule 

■  Typical  Class  Structure 

■  Hands-on  Projects 

■  Development  Details 

■  Summary  and  Take-aways 

■  What’s  Next 


Purpose  of  the  Presentation 


■  Discuss  experiences  developing  and  teaching  a  course  in  MBSE  with 
SysML 

■  Discuss  challenge  of  teaching  a  course  in  MBSE  with  SysML 

■  Discuss  course  background  information 

■  Discuss  techniques  employed  to  enhance  student  learning 


MBSE/SysML  Course  Challenge 


■  Develop  an  in-house  course  in  MBSE  with  SysML 

■  Goal:  Teach  concepts  as  well  as  practical  application 

■  Develop  an  effective  alternative  to  the  ‘all-day’  seminar 

■  Fire-hose  effect  -  too  much  info  to  absorb  in  a  short  period  of  time 

■  Good  for  overviews  but  not  enough  hands-on  learning 

■  Bottom-line 

■  Provide  students  with  training  needed  to  apply  SysML  concepts  and  the 
use  of  a  modeling  tool  to  their  current  projects 


Course  Background  Information 


Course  Background  Information 
-  Context 


■  MBSE 


-  MBSE  is  the  formalized  application  of  modeling  to  support  systems 
requirements,  design,  analysis,  verification,  and  validation  activities 
beginning  in  the  conceptual  design  phase  and  continuing  throughout 
development  and  later  life  cycle  phases.  [INCOSE,  Systems 
Engineering  Vision  2020,  Version  2.03,  Sept  2007] 


„ ,, 


SE  Practices  for  Describing  Systems 


OMG 

SYSTEMS 

MODELING 

LANGUAGE 


Afll* 


Past 


•  Specifications 

•  Interface 
requirements 

•  System  design 

•  Analysis  &  Trade-off 

•  Test  plans 


Future 


JEEEf* 

Moving  from  Document  centric  to  Model  centric 


4/15/2008 


Copyright©  2006-2008  by  Object  Management  Group. 


OMG 

incose  Model  Based  Systems  Engineering  ™|  •iz 

Benefits  K 

•  Shared  understanding  of  system  requirements  and  design 

-  Validation  of  requirements 

-  Common  basis  for  analysis  and  design 

-  Facilitates  identification  of  risks 

•  Assists  in  managing  complex  system  development 

-  Separation  of  concerns  via  multiple  views  of  integrated  model 

-  Supports  traceability  through  hierarchical  system  models 

-  Facilitates  impact  analysis  of  requirements  and  design  changes 

-  Supports  incremental  development  &  evolutionary  acquisition 

•  Improved  design  quality 

-  Reduced  errors  and  ambiguity 

-  More  complete  representation 

•  Supports  early  and  on-going  verification  &  validation  to  reduce  risk 

•  Provides  value  through  life  cycle  (e.g.,  training) 

•  Enhances  knowledge  capture 

4/1 5/2008  Copyright  ©  2006-2008  by  Object  Management  Group.  8 

- m — e~ 


slide  6 


Course  Background  Information 
-  Context 


SysML 

■  SysML  is  a  general  purpose  graphical  modeling  language  that  supports 
the  analysis,  specification,  design,  verification,  and  validation  of 
complex  systems.  [Friedenthal,  Moore,  and  Steiner,  A  Practical  Guide 
to  SysML,  p.  29] 


INCOSf 


Uli 


4  Pillars  of  SysML  -  ABS  Example 


OMG 

SYSTEMS 

MODELING 

LANGUAGE 


1.  Structure 

bdd  [Package]  Structure!  ABS  Structure  Hierarchy  ]J 


d  ABS_ActivationSequence [Sequence  Diagram] 


«block» 

Library.: 

Electronic 

Processor 


X 


«Hock» 

Anti-Lock 


z 


«block» 

Traction 

Detector 


definition 


«block» 

Library.: 


stmTireT faction  [State 


ibd  [Block]  Anti-Lock  Controller!  Basic  ]) 


C bd1;T 
- fcj  Det 


D  iagram] J 


act  Pre^ntLockup  [Activity  Diagram]  J 


JL  m1:B 
~H  Modu 


.X 

'9 


:  Modulate 
BrakingForce 


2.  Behavior 

interaction 

state 
machine 

activity/ 
function 


req  [Package)  Vehicle  Specifications  [  Braking  Requirements  ]J 

Vehicle  System  Specification!  Braking  Subsystem  Specification! 


«requirement» 

Stopping  Distance 


Id  =  "10.2“ 

Text  =  "The  vehicle  shall 
stop  from  60  miles  per  hour 
within  1 50  ft  on  a  clean  dry 
surface  * 

* 


«requrement» 

Anti-Lock  Performance 


id=‘33.r 

Text  =  The  braking  system  shall 
prevent  wheel  lockup  under  all 
braking  conditions.  “ 


-I- 


par  [Block]  Straight  Line  Vehicle  Dynamics  [  Parameters  ]J~ 


TJ  U  UrN 

7n — □ - 

el :  Braking  Force  . — 

— 1  e2 :  Acceleration 

Equation  | _ 

_ 1  Equation 

{f=(tf*bf)*(1  -tt)} 

a :  m/secA2 

□ 

e4 :  Distance  Equation 

□ 

{v-dx«t) 

e3 :  Velocity  Equation 

v :  m/sec 

v :  m/sec  [a=dv/dt] 

x: m  t:  seel 

n  t:  sec 

□ 

□ 

3.  Requirements 


4.  Parametrics 


18 
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Course  Background  Information 
-  Purpose 

■  Teach  MBSE,  SysML  concepts,  and  tool  use  to  JHU/APL  technical 
staff  members 

■  Introduce  Model  Based  Systems  Engineering 

■  Introduce  and  teach  SysML  concepts  and  techniques 

■  Demonstrate  and  teach  use  of  modeling  tool  to  produce  SysML  artifacts 

■  Motivation 

■  Increased  awareness  and  use  of  MBSE  and  SysML 

■  Application  of  concepts  to  projects 

■  Increase  staff  awareness  and  comfort  level  with  tool  usage 

■  Course  Objectives 

■  Learn  the  basics  of  MBSE  and  SysML 

■  Learn  the  basics  of  a  SysML-based  Tool 

■  Practice  application  of  basics  to  develop  system  models 
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Course  Background  Information 
-  Demographics 

■  Student  Information 

■  18  Students  (15  local,  3  remote) 

■  Systems  Engineering  background 

■  No  prior  MBSE  knowledge  required  or  assumed 

■  No  prior  SysML  or  UML  knowledge  required  or  assumed 

■  No  prior  SysML-based  tool  use  required  or  assumed 

■  Strategic  Education  Program  (SEP)  courses  at  JHU/APL 

■  Courses  for  JHU/APL  technical  staff  -  taught  by  JHU/APL  staff 

■  Non-credit 

■  Pass/Fail 


dPL 
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Course  Background  Information 
-  Text  and  Software  Used 

■  Course  Text 

■  “A  Practical  Guide  to  SysML:  The  Systems  Modeling  Language”; 
Friedenthal,  Moore  and  Steiner;  2008;  Elsevier,  Inc. 

■  Course  Software 

■  Sparx  Systems  “Enterprise  Architect”  (EA)  Ultimate  Edition  version  7.5 
(with  SysML) 

■  Academic  Licenses  for  instructors  and  students 

■  Instructor  experience 

■  Cisco  MeetingPlace 

■  Remote  student  participation 

■  Recorded  sessions  (presentations  and  voice) 

■  Microsoft  SharePoint 

■  Posting  Course  Material 


dPL 
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Course  Background  Information 
-  Coverage 


■  Three  things  required  for  modeling: 

■  Language 

■  Tool 

■  Methodology 

■  Focus  of  this  course  is: 

■  Language  (SysML) 

-  Tool  (EA) 


Methodology 


Language  Tool 


■  Several  MBSE  Methods  available 

■  Survey  of  Model-Based  Systems  Engineering  Methodologies  [INCOSE 
-TD-2007-003-01 ,  10  June  2008,  Estafan] 

■  SysML  and  EA  are  methodology-independent 

■  SysML  concepts  and  the  EA  tool  can  be  applied  to  various  MBSE 
methodologies 

■  Language  and  Tool  study  provide  the  foundation  for  Methodology  study 

■  Detailed  look  at  methodologies  -  good  candidate  for  follow-on  course 

■aiBfcm - dPL 
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Course  Background  Information 
-  Course  Schedule 


Week 

Date 

Hour 

Topic 

1 

9/8 

1&2 

Course  Overview,  Systems  Engineering  Overview,  Model  Based  Systems 
Engineering  Overview,  and  SysML  Overview 

2 

9/15 

1&2 

Organizing  the  Model  with  Packages  and  EA  Basics 

3 

9/22 

1&2 

Modeling  Requirements  and  their  Relationships 

4 

9/29 

1&2 

Motivation  for  MBSE  and  SysML 

5 

10/6 

1&2 

Modeling  Functionality  with  Use  Cases 

6 

10/13 

1&2 

Modeling  Structure  with  Blocks 
(Block  Definition  Diagrams) 

7 

10/20 

1&2 

Modeling  Flow-Based  Behavior  with  Activities 

8 

10/27 

1&2 

Modeling  Event-Based  Behavior  with  State  Machines 

9 

11/3 

1&2 

Modeling  Message-Based  Behavior  with  Interactions 

10 

11/10 

1&2 

Modeling  Structure  with  Blocks  (Internal  Block  Diagrams) 

11 

11/17 

1&2 

Modeling  Constraints  with  Parametrics 

12 

11/24 

1&2 

Modeling  Cross-Cutting  Relationships  with  Allocations 

^PL 
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Course  Background  Information 
-  Typical  Class  Structure 


1. 

2. 

3. 

4. 

5. 

6. 


Homework  Review 

Motivation: 

Language: 

Tool: 

Modeling  Example: 


Why  Model  <subject>  Diagrams  ? 

Concepts  (from  textbook) 

Using  EA  to  create  <subject>  Diagrams 
In-Class  Project  (automated  parking  garage  gate) 


Homework  Assignment 


Block  composition  can  be  depicted 
using  Composite  Associations 
Represents  the  Parts  that  make  up 
the  Whole 


•  Depicted  with  a  black  diamond  on 
the  Whole  end 

■  Multiplicity  on  the  Whole  end: 

■  Lower  bound  may  be  0  or  1: 

■  0  means  the  Part  can  exist 
without  the  Whole 

■  1  means  the  Part  always 
exists  within  the  Whole 

•  Upper  bound  is  always  1 

■  An  instance  of  a  Part  may 
exist  in  only  one  instance  of  a 
Whole  at  a  time 


■  Depicts ‘ownership’ 

•  Default  is  [0..1] 

Role  names  can  appear  on  the  part 
end  of  the  association 


API 


Language 


Tool  Model 


Uli 
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Sample  Language  Concepts  Slide 


■  Introduction  of 
SysML  elements 
and  relationships 
with  a  graphic 
example  of  each 


Composite  Association 


■  Block  composition  can  be  depicted 
using  Composite  Associations 

■  Represents  the  Parts  that  make  up 
the  Whole 


■  Depicted  with  a  black  diamond  on 
the  Whole  end 

■  Multiplicity  on  the  Whole  end: 

■  Lower  bound  may  be  0  or  1 : 

■  0  means  the  Part  can  exist 
without  the  Whole 

■  1  means  the  Part  always 
exists  within  the  Whole 

■  Upper  bound  is  always  1 

■  An  instance  of  a  Part  may 
exist  in  only  one  instance  of  a 
Whole  at  a  time 


©2008  Elsevier,  Inc.:  A  Practical  Guide  to  SysML 


■  Depicts ‘ownership’ 

■  Default  is  [0..1] 

■  Role  names  can  appear  on  the  part 
end  of  the  association 
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Sample  EA  Tool  Slide 


■  Step  by  step 
instructions  using 
EA  screen 
captures 

■  Simultaneous  EA 
tool  display 
demonstrating 
steps  using  EA 

■  Students  practice 
using  their  own 
laptops 
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Sample  Modeling  Example  Slide 


■  Combining  SysML 
concepts  and  tool 
usage  to  build  a 
SysML  artifact  for 
an  in-class  ‘real- 
world’  project 
system 


Block  Definition  Diagram  for  Gate  System 

bdd  Parkirtq  Garay  Gate  Components  ^ 


55 
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Course  Background  Information 

-  Hands-on  Projects 

■  Homework  Systems 

■  Alarm  Clock  Radio 

■  Coke  Machine 

-  Why? 

■  Familiarity 

■  Relatively  simple  systems  (as  compared  to  examples  in  text) 

■  Compare  and  contrast  student  models 

■  Practice 

■  Group  Homework  Projects 

■  Students  working  in  teams  on  homework 


Course  Background  Information 
-  Development  Details 

■  Course  Philosophy 

■  Need  to  practice  modeling  to  learn  it  -  course  needs  to  be  hands-on 

■  Minimalistic  approach 

■  Focus  of  course  is  on  the  basics  (not  complete  coverage) 

■  Just  enough  to  whet  the  appetite  without  being  overwhelming 

■  Learn-a-little  /  Practice-a-little  approach 

■  Two  hour  classes  /  once  a  week  /  twelve  weeks 

■  One  chapter  of  textbook  per  week 

■  Benefits 

■  Immediate  practice  of  learned  concepts 

■  Allows  one  week  of  practice  for  concepts  to  ‘sink-in’ 


dPL 
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Course  Background  Information 
-  Development  Details 

■  Section  Development  Process 

■  Create  ‘Reader’s  Digest’  version  of  a  chapter  from  the  text 

■  Extract  information  appropriate  for  an  Introductory  class 

■  Create  or  extract  graphics  to  illustrate  each  concept 

■  Create  SysML  Concepts  slides  using  information  from  book  and 
corresponding  graphics 

■  Create  EA  Tool  slides 

■  Develop  step-by-step  process  for  utilizing  the  SysML  concept  within 
the  EA  Tool 

■  Capture  EA  screens  in  order  to  ‘visualize’  the  process 

■  Create  slides  relating  process  steps  to  screen  captures 

■  Create  Modeling  Example  slides 

■  Apply  concepts  and  process  steps  discussed  to  a  real-world  system 

■  Create  slide(s)  capturing  model  depiction 


dPL 
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Course  Background  Information 
-  Development  Details 

■  Course  Material  Peer  Review 

■  All  course  material  was  presented  at  INCOSE  OOSEM  Working  Group 
meetings  for  review  and  comment 

■  INCOSE  OOSEM  Working  Group  consists  of  subject-matter  experts 
with  numerous  years  of  experience  in  Systems  and  Software 
Engineering,  MBSE,  UML,  and  SysML 

■  Includes  textbook  co-author  (Sandy  Friedenthal) 

■  Course  material  was  reviewed  incrementally  by  the  Working  Group 

■  Course  Outline 

■  Section  Development 

■  On-going  input  and  feedback  through  Course  Presentations 

■  Planned:  Contributions  to  post-course  improvements 


dPL 
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Summary  and  Take-aways 


MBSE/SysML  courses  should  include  adequate: 


Visual  learning  techniques  by  using  graphical  examples  of  language  concepts 
and  graphical  depictions  of  step-by-step  tool  usage  (visual  learning) 

In-class  instructor-lead  demonstrations  of  the  modeling  tool  (learning  through 
demonstrations) 

In-class  hands-on  training  with  a  modeling  tool  (learning  by  doing) 

Time  between  sessions  to  give  students  time  to  learn  and  practice  the  concepts 
outside  of  class  (incremental  learning  /  “sink-in”  time) 

Homework  projects  for  the  students  to  model  to  apply  the  concepts  that  they 
have  learned  to  sample  systems  (learning  through  practice) 

Group  homework  (collaborative  learning) 

Peer  review  of  course  matter  with  subject-matter  experts  (course  validation  and 
verification) 


Remotely  accessible  and  recordable  sessions  for  remote  (or  absent)  students 

(remote  learning) 
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What’s  Next 


■  Finish  teaching  current  course  (Nov  24th) 

■  Course  Evaluation 

■  Perform  Course  assessment 

■  Implement  improvements 

■  Develop  'Methodology’  Course  as  a  follow-on  to  this  Introductory 
course 

■  Investigate  offering  course  publicly  as  an  elective  in  the  Johns 
Hopkins  University  Engineering  for  Professionals  Master’s  Degree 
program  in  Systems  Engineering 


Questions 


- dPL 
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Update  of  the  Acquisition  Modeling 
and  Simulation  Master  Plan 

Stephen  J.  Swenson,  AEgis  Technologies  Group 

Acquisition  M&S  Community  Lead 
Systems  Engineering  Directorate 
Office  of  the  Director,  Defense  Research  and  Engineering 

12th  Annual  NDIA  Systems  Engineering  Conference 

October  29,  2009 


NDIA  SE  Conference:  DoD  AMSMP  Update 
10/27/09  Page-1 


UNCLASSIFIED 


Acquisition  M&S  Governance 

Structure 


Industry  DoD  Acquisition  DoD  M&S 


Mr.  Mike  Truelove  (Ctr) 

Chair.  Col  Eileen  Bjorkman,  USAF  Acquisition  Member: 

SAF/XCDM  OUSD(AT&L)/DDR&E/SE/MA 


AMSWG  Charter  (SE  Forum,  2006) 

•  Assist  PMs  and  acquisition  professionals  by  improving  the  utility  of  M&S  . . . 

•  Address  common  concerns,  improve  info  flow,  align  technical  initiatives,  pursue 
cross-cutting  issue  resolution  . . . 

•  Represent  the  acquisition  community  in  DoD  M&S  deliberations  . . . 
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UNCLASSIFIED 


Provide 
necessary 
policy  and 
guidance 


Current  AMSMP 


Objective  2  Objective  3  Objective  4 


Enhance  the 

■ 

Improve 

■ 

Improve 

■I 

technical 

model  and 

model  and 

framework 

simulation 

simulation 

for  M&S 

■ 

^capabilities 

1 

use 

K _ ^ 

■ 

Shape  the 
workforce 


1-1  M&S 

management 

1-2  Model-based 
systems 
engineering  & 
collaborative 
environments 

1-3  M&S  in  testing 

1-4  M&S  planning 
documentation 

1-5  RFP  &  contract 
language 

1-6  Security 
certification 


Key 

Broader  than  Acqn 


2-1  Product 
development 
metamodel 

2-2  Commercial 
SE  standards 

2-3  Distributed 
simulation 
standards 

2-4  DoDAF  utility 

a)  DoDAF  2.0 
Systems 
Engineering 
Overlay 

b)  Standards  for 
depiction  & 
interchange 

2-5  Metadata 
template  for 
reusable 
resources 
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3-1  Acquisition 
inputs  to  DoD 
M&S  priorities 

3-2  Best  practices 
for  model/sim 
development 

3-3  Distributed  LVC 
environments 

a)  Standards 

b)  Sim/lab/range 
compliance 

c)  Event  services 

3-4  Central  funding 
of  high-priority, 
broadly-needed 
models  &  sims 

a)  Prioritize  needs 

b)  Pilot  projects 

c)  Expansion  as 
warranted 
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4-1  Help  defining 
M&S  strategy 


5-2  Harvesting  of 
commercial 
M&S  lessons 


5-5  MSIAC  utility 

c)  Examination 
4-6  COTS  SE  tools 

4-7  M&S  in  acqn 
metrics 


5-3  Assemble  Body 
of  Knowledge 
for  Acqn  M&S 

5-4  M&S  education 
&  training 

a)  DAU,  DAG  & 
on-line  CLMs 

b)  Conferences, 
workshops  & 
assist  visits 


4-2  M&S  planning 
&  employment 
best  practices 

4-3  Foster  reuse 

a)  Business  model 

b)  Responsibilities 

c)  Resource 
discovery 

4-4  Info  availability 

a)  Scenarios 

b)  Systems 

c)  Threats 

d)  Environment 

4-5  W&A 

a)  Documentation 

b)  Risk-based 


5-1  Definition  of 
required  M&S 
competencies 


Circumstance 


•  Acquisition  Modeling  and  Simulation  Master  Plan  (AMSMP) 

-  Signed  out  April  17,  2006 

-  Forty  actions  designed  to: 

•  Foster  widely-needed  M&S  capabilities  beyond  the  reach  of  individual 
programs 

•  Better  enable  acquisition  of  effective  joint  capabilities  and  systems-of- 
sy  stems 

•  Empower  program  and  capability  managers  by  removing  systemic  M&S 
obstacles,  identifying  new  options  for  approaching  tasks,  and  helping 
support  widely-shared  needs. 

•  Promote  coordination  and  interface  with  M&S  activities  of  the  DoD 
Components. 

•  M&S  Steering  Committee  requires  that  each  community 
develop  and  maintain  a  business  plan 

•  Update  required  for  2010  to  feed  development  of  DoD’s 
Common  and  Cross-Cutting  Business  Plan 
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DoD  Modeling  &  Simulation 

Governance 


M&S  Management  Structure  Organized  by  Communities. 

Designed  to  Support  &  Integrate  M&S  Activities  across  the  Department. 

Led  by  a  1  to  2  Star  M&S  Steering  Committee  (M&S  SC)  to  provide  governance. 


Acquisition 

AT&L 


Analysis 

PA&E 

&J$ 


Experimentation 

JFCOM 


1  T 


Intelligence 

USD(I) 


I 


Planning 

JS 

&  Policy 


Testing 

DOT&E 
&  AT&L 


Corporate  &  Crosscutting  M&S  Tools 


Corporate  &  Crosscutting  M&S  Data 


Corporate  &  Crosscutting  M&S  Services 


)  (AP\EXC0M) 


Components 

OSD,  Joint  Staff,  COCOMs,  Services 


Goal:  Establish  corporate  M&S  management  to  address  DoD  goals: 
Leads/guides/shepherds  the  $Bs  in  DoD  M&S  investments;  adds  value 
thru  metrics  &  ROI-driven  priorities;  and  seeks  to  provide  transparency. 
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C&C  BP  Target  Timeline 
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6 


•  Description  of  the  “To-Be”  State  -  Vision 

•  Description  of  the  “As-ls”  State  -  Capabilities 

-  Tools 

-  Data 

-  Services 

•  Capability  Gaps 

•  Initiatives  /  Actions 

•  Acquisition  Community  M&S  Business  Plan 
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Update  AMSMP 


•  Completed  by  end  of  CY2009 

•  Current  AMSMP  is  our  departure  point 

-  Maintain  objectives 

-  Most  actions  will  carry-over,  update  as  required  to  reflect 
progress,  policy  change,  results  of  studies,  etc 

-  Completed  actions  will  be  replaced  by  their  follow-on  actions 

-  New  actions  will  be  added  to  reflect  change  in  business, 
policy  and  technology 

•  Structure  will  change 

•  Heavy  reliance  on  the  Acquisition  Modeling  and 
Simulation  Working  Group  (AMSWG) 
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End  Result 


*  Provide  cogent  guidance  for  choosing  projects  and 
influencing  other  acquisition  community  M&S 

-  Metrics  for  selecting  appropriate,  cost-effective  projects  traceable 
to  requirements 

-  Defined  interfaces  to  other  projects  and  activities 

-  Aligning  influence  on  other  acquisition  M&S 

*  Enable  systematic  integration  and  evaluation  of 
components  as  they  are  produced  &  assembled 

*  Allows  for  visible  progress  assessment  against  the 
vision,  holding  ourselves  accountable 

*  Provide  mechanism  for  iterating  requirements,  needed 
actions,  and  the  plan  accordingly 

*  Provide  guidance  for  influencing  policies  and  other’s 
activities 
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M&S  Activities  During  Acquisition 


MSA 


MSB 


MS  C 


MDD 


Materiel 

Solution 

Analysis 


Technology 

Development 


y 


Engineering  and 
Manufacturing 
Development 


y 


Production  and 
Deployment 

_ /X _ 


O&S  /// 


PDR  or  PDR 


CDR 


Full  Rate  Production 
Decision  Review 


Develop  M&S  requirements 
(SEP) 

Model  and  data  discovery 
(SEP) 

Develop  M&S 

Configuration  Management 
Strategy  (SEP) 

Assign  Lifecycle 
Maintenance 
Responsibilities  of  Data 
and  Models  (SEP,  RFP) 
Define  role  of  M&S 
throughout  the  lifecycle 
Review  and  identify 
appropriate  standards  for 
M&S  reuse  and 
interoperability  (SEP,  RFP) 
Develop/Modify  Required 
M&S 

Accredit  applicable  M&S 
capabilities 

Publish  descriptions  of 
M&S  capability  developed 
in  this  phase. 

Asses  required  data 
ownership/use  rights  and 
accessibility  (development 
RFP,  materiel  RFP) 


Develop  M&S  requirements 
(SEP) 

Influence  Acquisition 
Strategy  to  address 
modeling  and  simulation 
requirements  and  use. 
Document  role  of  M&S  in 
testing  and  initiate 
identification  of  required 
M&S  assets 
Initiate  discussion  of 
requirements  for  use  of 
M&S  in  operational  test 
w/OT  community  (TEMP) 
Update  SEP  based  on 
evolving  M&S  requirements 
Develop/modify  required 
M&S  including  virtual 
prototype 

Accredit  applicable  M&S 
capabilities 

Publish  descriptions  of  M&S 
capability  developed  in  this 
phase. 

Review  data  and  ownership 
rights  (materiel  RFP) 


Develop  M&S 
requirements  (SEP) 
Develop/modify 
required  M&S 
Accredit  applicable 
M&S  capabilities 
Publish  descriptions 
of  M&S  capability 
developed  in  this 
phase. 

Review  data  and 
ownership  rights 
(LRIPRFP) 


Identify  opportunities 
for  M&S  reuse  for 
operations  (e.g. 
training,  decision 
support,  etc) 
Develop/modify 
required  M&S 
Accredit  applicable 
M&S  capabilities 
Publish  descriptions 
of  M&S  capability 
developed  in  this 
phase. 

Review  data  and 
ownership  rights  (Full 
Rate  RFP) 


Capture  data  to 
strengthen  M&S  for 
operational  use  and 
feedback  to  other 
programs. 

Reuse/repurpose  M&S 
for  operational  use 
(e.g.,  training, 
decision  support,  etc) 
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Early  Involvement  (Pre-Milestone  A) 
Will  Reduce  Total  Ownership  Cost 


New 

5000.02  / 
WSARA 
Approach 


f  Serial  i 
Approach 


NJ 


Materiel 

Solution 

Analysis 


Engineering  & 
Manufacturing 
Development 


Technology 

Development 


Production  & 
Deployment 


Operations 
&  Support 
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Acquisition  Lifecycle  Comparisons 

Defense  Acquisition  Management  System,  May  12,  2003 


A 


A 


(Program  Initiation) 


E 


Concept 

Technology 

System  Development  and 
Demonstration 

Production  and  Deployment 

Operations  and 

Refinement 

\ 

Development 

Design 

\y  Readiness 

Review 

*  Full-Rate 
\y  Production 
Decision 

Support 

r 

Concept 

Decision 


Review 


Defense  Acquisition  Management  System,  December  8,  2008  (new  DODI  5000.02) 

A  A  (Program  Initiation)  A 


Materiel 

Solution 

Technology 

Engineering  and  Manufacturing 
Develnnment 

Production  and  Deployment 

Operations  and 
Support 

.  Analysis 

L _ _ _ 

(PDR) 

(PDR.i  (CDR)  ^ 

/\  FRP 
Decision 

v  Review 

Development 

Decision 


Or 


PDR  after  B 
w/  Post-PDR 
Assessment 


Post-CDR 

Assessment 


Defense  Acquisition  Management  System,  May  22,  2009  (WSARA) 

A  Technology  A  A 

Development  /|-\  (Program  Initiation)  /  C  \ 


4 


Materiel 

Solution 

Analysis 


Materiel 

Development  Decision 


Technological  Maturity 
and  Integration  Risk 
Assessment 

(PDFJ)^X 


Engineering  and  Manufacturing 
Development 


Competitive 

Prototyping 


0 


Production  and  Deployment 

a  FRP 

\y  Decision 
Review 


Operations  and 
Support 


*ost-PDR 
Assessment 


Post-CDR 

Assessment 
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Acquisition  M&S  Master  Plan 
Update  Process 


(Top-down) 

Desired  Acqn  Environment  per 
CJCSI  3170,  DoDD  5000.01, 
_ WSARA _ 


Identify  Needed  System 
Engineering  Capabilities 


Jan  ‘10 


Acquisition  M&S 
Master  Plan 


Determine  &  Prioritize  What 
Acqn  Community  Must  Do 


Identify  Actions  of  Others 
(e.g.,  M&S  CO,  Nil,  NIST) 


Dec  ‘09 


Identify  Actions  Needed 
to  Address  the  Gaps 


Nov  ‘09 


Identify  M&S  Capability  Gaps 


Sep-Oct  ‘09 


Identify  Needed 
M&S  Capabilities 


Assess  Current  Issues/Needs 
(e.g.,  Systems  of  Systems  efforts) 


(Bottom-up) 


Assess  Recommendations  fm 
Prior  M&S  in  Acqn  Studies 
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Policy/Guidance  Reference  Documents 


1  .DoD  Directive  5000.1,  “The  Defense  Acquisition  System,”  May  12,  2003 

2. DoD  Directive  5000.59,  “DoD  Modeling  and  Simulation  (M&S)  Management,”  August  8,  2007 

S.Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  31 70.01  G,  "Joint  Capabilities  Integration  and 
Development  System,"  March  1,  2009 

4. DoD  5025.1 -M,  "DoD  Directives  Systems  Procedures,"  October  28,  2007 

5.DoDD  8320.2,  “Data  Sharing  in  a  Net-Centric  Department  of  Defense,”  December  3,  2004 

6. DoD  5000.59-M,  "Glossary  of  Modeling  and  Simulation  Terms,"  January  15,  1998 

7. Defense  Acquisition  University,  "Glossary  of  Acquisition  Acronyms  and  Terms,"  July  2005 

8.“Defense  Acquisition  Guidebook,  Version  X.Y,”  November  1,  2006 

9. DoD  Instruction  5000.2,  “Operation  of  the  Defense  Acquisition  System,”  Dec  8,  2008 

10.“Federal  Acquisition  Regulation,”  March  31,  2008 

1 1  .DoD  Instruction  8500.2,  “Information  Assurance  (IA)  Implementation,”  February  6,  2003 

12.“DoD  Architecture  Framework,”  April  23,  2007 

IS.DoDI  5000.61,  “DoD  Modeling  and  Simulation  (M&S)  Verification,  Validation,  and  Accreditation 

(W&A),”May  13,  2003 

14.  Revision  to  T&E  Policy;  Memorandum;  December  22,  2007 

15.  DoD  M&S  Human  Capital  Strategy  (DRAFT) 

16.  Implementing  a  Life  Cycle  Management  Framework;  DTM;  July  31,  2008 

1 7.  Weapons  Systems  Acquisition  Reform  Act;  May  22,  2009 
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Needed  Systems  Engineering 

Capabilities 


SE1  Early,  continuing  systems  engineering  from  an  SoS/FoS 
capabilities  perspective;  seamless  transition  from  JCIDS 
to  acquisition 

SE2  Lifecycle-wide  exploration  of  the  maximum  available  trade 
space,  including  time-phased  requirements  and 
technology  insertion 

SE3  Collaboration  among  multiple  organizations,  Service  & 
contractors  for  all  key  enterprise-level  SE  decisions 

SE4  Comprehensive,  accurate,  early  assessment  of  designs; 
avoidance  of  costly  fixes  for  problems  discovered  late  in 
the  acquisition  process 

SE5  Tighter  decision  cycles  (faster  design-assessment 
process) 

SE6  More  effective  &  efficient  testing,  including  in  a  SoS/FoS 
environment 

SE7  Appropriate  reuse  of  all  resources  -  information,  software 
tools,  expertise,  facilities,  ranges,  etc  --  across  programs 
&  organizations 
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M&S  Processes  for  Better  Systems 

Engineering 


MSI  Use  of  a  model-based  engineering  approach 

MS2  Establishing  M&S-enabled  collaborative  engineering 
environments 

MS3  Model-Test-Model  process  to  improve  both  M&S  tools  and 
testing 

MS4  Harnessing  M&S  knowledge  to  formulate  an  effective  M&S 
strategy 

MS5  Disciplined  M&S  planning  and  employment 

MS6  Efficient  development/maintenance  of  credible  M&S  tools 

MS7  Access/sharing  of  authoritative  data  needed  for  M&S 
representations 

MS8  Inspection  of  M&S  used  and  cost  burden  that  inhibits  M&S 
use 
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A  |  B  0  |  D  |  E 

F 

G 

H  1  J 

1 

MtS  PROCESS  DESCRIPTION 

CAPABILfTT(IES) 

GAP 

METRIC 

£ 

MSI 

Urf-  of  a  Model-Based  EnqinL-i>rinq 
(MBE)  Approach 

Individual  acquirition  proqramr  are  beqinninq  to 
employ  MSBE  throuqh  vendor  offerinqr. 

Number  of  acquisition  proqrams  uhich 
have  employed  MSBE  (up-qood). 
Number  of  unique  MSBE  approaches 

3 

MS1.1 

F'  >j  b- 1  Lr  hi  l-  -J  Model-Based 
Enqineerinq  -3  p-  p-  r  □  -3  ■:  K  T  l-x  j . 

Vendors  ofsystems  enqineerinq  tools  have 
develop  MSBE  approaches.  IMCCSEMSBE 
MethodoloaY  Survey  articulatesseveral  extant 

The  DoD  has  not  adopted  a  particular  MSBE 
approach(es). 

4 

MSI. 1.1 

Approachfe/Jforthe 
-3  e-  e-  1  i  -3  h  i  □  n  of  M&S  ha 

5 

MS1.1.£ 

Approach(es) for  the 

■3  e-  c- 1  i  ■:  -3  h  i  □  n  of  M&S  ha  Systems 

t 

MSI. 1.3 

Approach(es)for  the 

-3  e-  c- 1  i  -=  -3  •:  i  a  n  □  H  M  :i=:  S  l-t  i  -3  r. 

T 

MSI. 1.4 

Approachfe/Jforthe 

-3  e-  c- 1  i  -3  h  i  □  n  of  M&S  ha  Test  and 

$ 

MS1.£ 

An  acquisition  uorkforce  that 
understands  the  principle  and 
value  of  a  Model-Based  Systems 

* 

MS1.£.1 

Body  of  Knouledqe  for  Model- 
Based  Systems  Enqineerinq. 

SimSummit  Body  of  Knouledqe.  AFAMS  BoK 
outline.  IMCOSE  Systems  Enqineerinq  BoK. 

Current  M&S  BoK  lackssiqnificant  content 
describinq  the  principles  and  value  of  MSBE. 

G34  Bb-Jj  if  KbbuI*-I-i*  far  HtS 

10 

MS1.£.£ 

Educational  opportunities 
provided  by  qovernment  and 
academia. 

11 

MS1.£.£.1 

Education  for  acquisition 
proqram  manaqers  on  the  value 
and  application  of  MBSE 

Courseuork  developed  under  the  'Educatinq  the 
Acquisition  Workforce1  hiqh  level  task" 
available  throuqh  Naval  Past  Graduate  School. 
M&S  for  Acquisition  Continuous  Learninq 

Module  (CLM)  Test  and  Evaluation  CLM  offered 
throuqh  the  Defense  Acquisition  University. 

Current  educational  opportunities  lack 
siqnificantMSBE content. Gtt  Hb 

cflutuir  at  tie  *al»  af 
iittqrattd  arckilectirts,  bbi 
respiisibilitj  far.  G4I  Acq> 
cflBBHity  aaBaqftrs  aidstaffi 
Birtlj  iiikfBrBtd  al-BBt  HtS 
l  j  zm.  c 

Number  of  acquisition  PM-track 
professionals  and  military  personnel 
enrolled  in  qouernment-sponsored 
courses  that,  as  a  minimum,  address  the 
iiflliiAiifM^RF  f,, . A\ 

1£ 

MS1.£.£.£ 

Education  for  the  qeneral 
acquirition  professional 
includina  JY/temr  enaineen. 

Students  enrolled  in  qovernment- 
spDnsoredMSBEcDurses.  (up-qood) 

13 

MSI. £.£.3 

Education  for  the  modelinq  and 
/Emulation  professional  uith 
particular  empharir  on  buildinq 
capability  for  reure  across 
lifecycle,  orqanixational  and 
corporate  boundaries. 

Courseuork  developed  under  the  'Educatinq  the 
Acquisition  Workforce'  hiqh  level  task  — 
available  throuqhseveral  universities.  Other 
Universities  are  beqinninq  to  off er  deqrees  in 
modelinq  andsimulation  (e.q.,  Old  Dominion, 
Georqia  Tech,  U.  Alabama  Huntsville) 

Some  U.S.  colleqe/f  universities  offer  either 
courseuork/seminars  dedicated  to  MSBE  or 
apply  MSBE  to  other  academic  ends. 
Examples  include:  University  of  Michiqan 
of f ersseminars  on  MSBE.  MIT  offers  a 
professionalshortcourse  entitled:  'Systems 
Enqineerinq,  Architecture  and  Lifecycle 
Desiqn:  Principles,  Models,  Tools,  and 
Applications.'  Boston  University  offers 
courses  in  modelinq  andsimulation  uithin 
their  Division  of  Systems  Enqineerinq  i 
Colleqe  of  Enqineerinq.  INCCSE  maintains  a 
directory  of  academic  institutions  that  offer 
educational  opportunities  in  Systems 
Enqineerinq: 

Number  of  colleqesfuniversities 
offerinq  instruction  in  MSBE.  (up-qood) 

14 

MSI. 3 

Personnel  competence- 

15 

MSI. 3.1 

Standard/  of  professional 
competence  in  MSBE. 

Certification  criteria  for  modelinq  and 
simulation  professionals  as  reflected  in  the 

NTS  A  Certified  Modelinq  and  Simulation 
Professional  proqram.  Systems  Enqineerinq 
professional  competecencies  are  under 
development  and  are  expected  to  be  complete  in 
.:i  -  mHm  r a . l  ..  lL  .  i .  .  j> 

Need  coqentstandards  for  professional 
competence  usinq  MSBE  based  on  Systems 
Enqineerinq  knouledqe, skills  and  abilities. 

Standards  exist. 

10 

MS1.3.£ 

Prof e/jinnal  certification  for 

Civil  Servant/ 

DAWIA  (Systems  Enqineerinq,  Proqram 
Manaqement),  Certified  M::::S  Professional 
fNTSAl.  Heavily  influenced  by  M&S  for  DoD 

Current  certifications  are  for  qeneral 
modelinq  andsimulation.  No  certification  for 
MSBEcomc-etence. 

Number  of  certified  professionals  both 
in  and  out  of  qovernment.  (up--qood). 
Number  of  ueaponsystems  acquisitions 

1 

_ 

r-..i.:e:.  j kh-xs c._e . -umtsa^  u.  ...:i.. 

hi.  .  j  .hi— a  i. 

.  J 

i 


H  i  ►  M  \MSl/  MS2  /  MS3  /  MS4  /  MS5  /  MS6  /  MS7  /  MSS 
Ready 


i< 


SU 


E 
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Gap  Examples 


•  Body  of  Knowledge  for  M&S  support  to  acquisition  is 
deficient,  not  managed — lacks  specific  guidance  for  MBE. 

•  The  DoD  has  not  adopted  a  particular  MBE  approach(es). 

•  No  DoD  requirement  for  formal  M&S  planning  to  support 
acquisition  (other  than  T&E). 

•  Need  ability  to  identify  competent  personnel  in  industry 
offerings  to  the  government. 

•  An  overabundance  of  capabilities  that  essentially 
accomplish  the  same  thing  leads  to  an  almost 
indecipherable  landscape.  Need  to  focus  attention  in  one 
direction  and  merge  capabilities  from  the  others. 

•  Use  of  DoD-unique  standards  limits  their  user  base,  quality, 
COST  tool  support,  and  opportunities  for  reuse. 

•  The  average  producer  of  modeling  and  simulation 
capability  has  little  knowledge  of  the  existence,  value  or 
use  of  currently  available  discovery  metadata 
specifications. 
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Business  Ecosystem 


“An  economic  community  supported  by  a  foundation  of 
interacting  organizations  and  individuals  -  the  organisms  of  the 
business  world.  This  economic  community  produces  goods  and 
services  of  value  to  customers,  who  themselves  are  members  of 
the  ecosystem.  The  member  organizations  also  include  suppliers, 
lead  producers,  competitors  and  other  stakeholders.  Overtime, 
they  coevolve  their  capabilities  and  roles,  and  tend  to  align 
themselves  with  the  directions  set  by  one  or  more  central 
companies.  Those  companies  holding  leadership  roles  may 
change  over  time,  but  the  function  of  ecosystem  leader  is  valued 
by  the  community  because  it  enables  members  to  move  toward 
shared  visions  to  align  their  investments  and  to  find  mutually 
supportive  roles.” 

James  F.  Moore,  The  Death  of  Competition  - 
Leadership  and  Strategy  in  the  Age  of  Business 
Ecosystems,  Harper  Business,  New  York,  1996. 
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Measures  to  Assess  an 
Ecosystem’s  Health 


•  Productivity 

-  The  ability  of  the  ecosystem  to  continually  transform 
technology  and  raw  materials  of  innovation  into  lower  costs 
and  new  products 

•  Robustness 

-  An  ecosystem’s  ability  to  survive  major  disruptions,  such  as 
those  caused  by  unpredictable  technological  innovation  and 
change 

•  Niche  Creation 

-  Ability  of  an  ecosystem  to  increase  meaningful  diversity 
through  the  creation  of  valuable  new  functions,  or  niches 


lansiti,  Marco  and  Levien,  R;  “Strategy  as 
Ecology”,  Harvard  Business  Review,  March  2004 
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Principles  of  the  Ecosystem  Model 


•  “Open  system”:  organic  systems  exist  in  a  continuous  exchange  with  their 
environment,  characterized  by  a  continuous  cycle  of  input,  internal 
transformation  (throughput),  output,  and  feedback. 

•  Homeostasis:  self-regulation  and  the  ability  to  maintain  a  steady  state  achieved 
through  processes  that  regulate  and  control  system  operation  on  the  basis  of 
“negative  feedback”  whereby  deviations  from  some  standard  norm  initiate 
actions  to  correct  the  deviation. 

•  Entropy/negative  entropy:  closed  systems  are  entropic  in  that  they  have  a 
tendency  to  deteriorate  and  run  down.  Open  systems  seek  to  sustain 
themselves  by  importing  energy  -  they  are  characterized  by  negative  entropy. 

•  Structure,  function,  differentiation,  and  integration:  relationship  between  these 
concepts  is  crucial  to  understanding  living  systems  as  they  are  closely 
intertwined. 

•  Reguisite  variety:  the  internal  regulatory  mechanisms  of  a  system  must  be  as 
diverse  as  the  environment  with  which  it  is  trying  to  deal. 

•  Equifinality:  in  an  open  system,  there  may  be  many  different  wavs  of  arriving  at 
a  given  end  state. 

•  System  evolution:  the  capacity  of  a  system  to  evolve  depends  on  an  ability  to 
move  to  more  complex  forms  of  differentiation  and  integration.m 

Ml  Gareth  Morgan,  Images  of  Organization. 
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2012  Revision  of  NAICS 


•  Office  of  Management  and  Budget  (OMB)  Federal  Register  notice  soliciting 
proposals:  Late  2008/Early  2009 

-  Released  Jan  7  with  proposals  due  April  7 

•  U.S.  Economic  Classification  Policy  Committee  (EPCP)  review  of  proposals  and 
trilateral  negotiation:  ongoing  through  2009 

•  Federal  Register  notice  containing  ECPC  recommendations  to  OMB:  late  2009 
or  early  2010 

•  Federal  Register  notice  containing  OMB  final  decisions:  May  2010 

•  2012  NAICS  United  States  Manual  manuscript  submitted  to  OMB:  June  2011 

•  2012  NAICS  United  States  Manuals  available:  January  2012 
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Questions? 


NDIA  SE  Conference:  DoD  AMSMP  Update 
10/27/09  Page-23 


UNCLASSIFIED 


Technical  Reference  Documents 


1.  Final  Report  of  the  Acquisition  Task  Force  on  M&S,  1994;  Sponsor:  DDR&E  (Dr.  Anita  Jones);  Chair: 
VADM  T.  Parker,  USN  (Ret.) 

2.  Naval  Research  Advisory  Committee  Report  on  M&S,  1994;  Sponsor:  ASN(RDA);  Chair:  Dr.  Delores  Etter 

3.  Collaborative  Virtual  Prototyping  Assessment  for  Common  Support  Aircraft,  1 995;  Sponsor:  Naval  Air 
Systems  Command;  conducted  by  JHU/APL  and  NSMC 

4.  Collaborative  Virtual  Prototyping  Sector  Study,  1996;  North  American  Technology  &  Industrial  Base 
Organization;  sponsor:  NAVAIR 

5.  Application  of  M&S  to  Acquisition  of  Major  Weapon  Systems,  1996;  American  Defense  Preparedness 
Association;  sponsor:  Navy  Acqn.  Reform  Exec. 

6.  Effectiveness  of  M&S  in  Weapon  System  Acquisition,  1996;  Sponsor:  DTSE&E  (Dr.  Pat  Sanders); 
conducted  by  SAIC  (A.  Patenaude) 

7.  Technology  for  USN  and  USMC,  Vol.  9:  M&S,  1997;  Naval  Studies  Board,  National  Research  Council; 
sponsor:  CNO 

8.  A  Road  Map  for  Simulation  Based  Acquisition,  1998;  Joint  SBA  Task  Force  (JHU  APL  lead);  sponsor: 
Acquisition  Council  of  EXCIMS 

9.  M&S  for  Analyzing  Advanced  Combat  Concepts,  1999;  Defense  Science  Board  Task  Force  (Co-chairs:  L. 
Welch,  T.  Gold) 

10.  Advanced  Engineering  Environments,  1999;  National  Research  Council;  sponsor:  NASA 

11.  Survey  of  M&S  in  Acquisition,  1999  and  2002;  Sponsor:  DOT&E/LFT&E;  conducted  by  Hicks  & 
Associates  (A.  Hillegas) 

12.  Test  and  Evaluation,  1999;  Defense  Science  Board  Task  Force  (Chair:  C.  Fields) 

13.  “SIMTECH  2007”  Workshop  Report,  2000;  Military  Operations  Research  Society  (Chair:  S.  Starr) 

14.  M&S  in  Manufacturing  and  Defense  Systems  Acquisition,  2002;  National  Research  Council;  sponsor: 
DMSO 

15.  M&S  Support  to  the  New  DoD  Acquisition  Process,  2004  NDIA  Systems  Engineering  Div.  M&S 
Committee;  sponsor:  PD,  OUSD(AT&L)DS 

16.  Missile  Defense  Phase  III  M&S,  2004  Defense  Science  Board  Task  Force  (Chair:  W.  Schneider) 

17.  Live,  Virtual,  Constructive  Architecture  Roadmap,  2008,  JFCOM  (Lead:  K  Goad) 

18.  Modeling  and  Simulation  Resource  Reuse  Business  Model,  2008,  Center  for  Naval 
Analyses  (Lead:  D.  Shea) 
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M&S  Activities  During  Acquisition 


MSA 


MSB 


MS  C 


Materiel 
Solution 
MDD>  Analysis 


Technology 

Development 


y 


Engineering  and 
Manufacturing 
Development 


y 


Production  and 
Deployment 

_ /X 


PDR  or  PDR 


CDR 


Full  Rate  Production 
Decision  Review 


Develop  M&S  requirements 
(SEP) 

Model  and  data  discovery 
(SEP) 

Develop  M&S 

Configuration  Management 
Strategy  (SEP) 

Assign  Lifecycle 
Maintenance 
Responsibilities  of  Data 
and  Models  (SEP,  RFP) 
Define  role  of  M&S 
throughout  the  lifecycle 
Review  and  identify 
appropriate  standards  for 
M&S  reuse  and 
interoperability  (SEP,  RFP) 
Develop/Modify  Required 
M&S 

Accredit  applicable  M&S 
capabilities 

Publish  descriptions  of 
M&S  capability  developed 
in  this  phase. 

Asses  required  data 
ownership/use  rights  and 
accessibility  (development 
RFP,  materiel  RFP) 


Develop  M&S  requirements 
(SEP) 

Influence  Acquisition 
Strategy  to  address 
modeling  and  simulation 
requirements  and  use. 
Document  role  of  M&S  in 
testing  and  initiate 
identification  of  required 
M&S  assets 
Initiate  discussion  of 
requirements  for  use  of 
M&S  in  operational  test 
w/OT  community  (TEMP) 
Update  SEP  based  on 
evolving  M&S  requirements 
Develop/modify  required 
M&S  including  virtual 
prototype 

Accredit  applicable  M&S 
capabilities 

Publish  descriptions  of  M&S 
capability  developed  in  this 
phase. 

Review  data  and  ownership 
rights  (materiel  RFP) 


Develop  M&S 
requirements  (SEP) 
Develop/modify  required 
M&S 

Accredit  applicable  M&S 
capabilities 

Publish  descriptions  of 
M&S  capability 
developed  in  this  phase. 
Review  data  and 
ownership  rights  (LRIP 
RFP) 


Identify  opportunities  for 
M&S  reuse  for 
operations  (e.g.  training, 
decision  support,  etc) 
Develop/modify  required 
M&S 

Accredit  applicable  M&S 
capabilities 

Publish  descriptions  of 
M&S  capability 
developed  in  this  phase. 
Review  data  and 
ownership  rights  (Full 
Rate  RFP) 


Capture  data  to 
strengthen  M&S  for 
operational  use  and 
feedback  to  other 
programs. 

Reuse/repurpose  M&S 
for  operational  use  (e.g., 
training,  decision 
support,  etc) 
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M&S  Use  During  Acquisition 


MSA 


MSB 


MS  C 


Materiel 
Solution 
MDD>  Analysis 


Technology 

Development 


y 


Engineering  and 
Manufacturing 
Development 


y 


Production  and 
Deployment 

_ /X 


PDR  or  PDR 


CDR 


Full  Rate  Production 
Decision  Review 


AoA 

Rapid  virtual 
prototyping 
Exploration  of 
alternatives  and  design 
variations 
CAD/CAM 

Promote  stakeholder 
inspection  of  proposed 
solution,  variations  and 
alternatives 

Identify  cost  drivers  and 
risk  areas. 

Conduct  initial 
manpower 

requirements  studies. 


System  performance 
analyses  (e.g.,  evaluate 
Pdet,  Pcded,  Pk,  etc) 
CAD/CAM 
Human  machine 
interface  design 
Failure  analyses  (e.g., 
stress,  fatigue,  shock) 
Conduct  manpower 
requirements  studies. 
Evaluate  cost 
implications 
Life  cycle  cost  analyses 
Analyze  and  assess 
resource,  readiness,  and 
other  key  life  cycle 
sustainment  metrics 


Evaluate  performance  of 
technology  under 
development. 

Focus  test  and 
evaluation  activities 
CAD/CAM 

Test  and  evaluation 
under  conditions 
otherwise 

difficult/impossible  to 
replicate  (i.e.,  safety 
restrictions, 
environmental 
restrictions,  cost 
restrictions) 

Predict  human 
performance  as  a 
function  of  detailed 
design 

Anthropometry  and 
biomechanics. 

Analyze  and  assess 
resource,  readiness,  and 
other  key  life  cycle 
sustainment  metrics 


Design  of  manufacturing 
facilities 

Define  production 
workflow 

Analyze  and  assess 
resource,  readiness,  and 
other  key  life  cycle 
sustainment  metrics 


Support  design  and 
maintenance 
modifications 
Evaluate  redesign 
efforts 

Analyze  and  assess 
resource,  readiness,  and 
other  key  life  cycle 
sustainment  metrics 


NOTE:  THIS  LIST  IS  NOT  COMPLETE  AND  GROUPING  BY  PHASE  IN  THIS  WAY  DOES  NOT  ADEQUATELY  COMMUNICATE 

BROAD-SPECTRUM  USE  M&S  THROUGHOUT  THE  ACQUISITION  PROCESS 
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Mentioned/implied  in 
the  definition  of  the 
=|  CDD  (evolutionary 
acquisition) 


To  CJCSI  31 70.01  G 
dtd.  1  Mar  2009 


KeyCiSCI  3170.01  E  Policies 

*3r 


Joint  conceots-centhc  capabilities  identification  process  to  allow  joint 
forces  to  meet  the  full  range  of  military  operations  and  challenges... 
Assess  existing  and  proposed  capabilities  in  light  of  their  contribution 
to  future  joint  allied  and  coaliti^Joperations. ...  Produce  capability 
proposals  that  consider  the  fulirandibf  DOTMLPF  solutjons  in  order 
to  advance  joint  warfighting  in  a  unilateral  and  multinational  context. 
New  solution  sets. .  .crafted  to  deliver  technologically  sound .  testable 
AE3  sustainable  and  affordable  increments  of  militarily  useful  capability. 

•  The  FgS  andffiS  solutions  may  also  'require  systems  delivered  by 
multiple  spon  ;ors/matenel  deyejggers^Es 
■  The  process  i  a  identify  capability  gaps and  potential  solutions  must  be 
supported  by  a  robust  analytical  Q/vcess  aes 
JCIDS  implen  ents  a  capabiliti&fbased  apftpach  that. .  .requires  a 
'  AE?  collaborative  j  process  that  utMzes  joint  concms  and  integrated 


E8 


architectures .  o  identify  piwritized  capability  gh>$  and  integrated 
1  policy  approaches  to  resolve  th\e  gaps 


Mentioned  only  in 
the  definition  of 
“Materiel  Solution” 


Implied 


throughouT 
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AMSMP  Update  Database 


Systems  Engineering 
Artifacts;  Fully 
Traceable. 


Master  Plan 
Contents 


PJS. 

DesAcq  Env _ . — 

•e^fTp 

Designator 

Description 

DesAcqEnvColl 
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S] 
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SEC  ID 
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MSProcesses 

PK 
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Description 
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PK 
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Description 

Gap_Action_MAP 

FK1 

ActionDesionatef- 

PlanObjgplt<re 

tTbiectivelP 
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ActionStatusVI 

PK 

ActionStatusID 

Action  Designator 

ActionProgress 

NextSteps 

Description 
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Discussion 

Products 

CompletionGoal 

ParentObjective 

ChangeNotes 
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PK 

MemberlD 

FK1 
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FirstName 
PhoneNumber 
Email  Add  r 
OfficeDesignator 

PreliminaryVote 

FK1 

FK2 

AMSWGMember 
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Vote 

Comment 
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PK 

Officehasianator 

Mission 

LeadResource 

PK.FK2 

OfficeDesignator 

FK1 

ActionDesignator 

ChangeNotes 

SupportlResource 

PK.FK2 

OfficeDesianator 

FK1 

ActionDesignator 
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Reference 

Document 

- ^ 

PK 

DocumentID 

FK1 

FK2 
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DocumentID 

IssuanceRecord 
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Signature^afe 
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AMSMP  Traceability  Map 


- 5E 

LABILITIES 


M&S 

PROCESSES 


GAP 


ACTIONS 
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Ogden  Air  Logistics  Center 


KIHOMAC 


A-10  Avionics 
System 

Architecture  Trade 
Analysis 

(AVSATA)  Program 

Jerry  Coates 
A-10  OSS&E  Integrator 
OO-ALC/538  ACSG/EN 

Richard  Sorensen 
KIHOMAC  Staff  Systems  Engineer 


System  Acquisition  Excellence 


Agenda 


OGDEN  AIR  LOGISTICS  CENTER 


■  A-10  Background 

■  Architecture  &  Requirements  Overview 

■  A-10  Architecture  Development 

■  Example 

■  Path  Forward 

■  Results 


KIHOMAC 


System  Acquisition  Excellence 


OGDEN  AIR  LOGISTICS  CENTER 


A-10  BACKGROUND 


KIHOMAC 


System  Acquisition  Excellence 


Legacy  Aircraft 

The  “Green  Machine” 


OGDEN  AIR  LOGISTICS  CENTER 


A-10  designed  as  a  tank  buster,  low-technology,  easy  to 
maintain  ground  attack  fighter 

■  A-1 0  upgrades  limited  in  scope  and  capability. 

■  Sustainment  programs 

•  Largely  form/fit/function  replacements. 

■  Lack  of  funding  and  a  master  plan  (architecture  roadmap)  resulted  in 
stovepipe  sustainment/capabilities  modifications  without 
considerations  for: 

•  Systems  Engineering 

•  Distribution  of  functions 

•  Growth  of  capabilities 

•  Interoperability 


KIHOMAC 


System  Acquisition  Excellence 


Beyond  Design  Life 


OGDEN  AIR  LOGISTICS  CENTER 


Notional 

Projected  Lifetime 


50 


Years 


Ex  ended  Life 


11)0 


f~|  Base  Model  Program  Start  Q  Base  Model  IOC  Q  Planned  Phase  Out 

(  asc  Model) 

Baseline  Graph  extracted  from  USAF  Viable  Combat  Avionics  Initiative  Implementation,  Mr.  Doug  Ebersole; 
AFMC  Aeronautical  Enterprise  Program  Office;  22  Oct  02;  pg  5 
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System  Acquisition  Excellence 


Precision  Engagement 

This  Ain’t  Your  Daddy’s  Hog 


OGDEN  AIR  LOGISTICS  CENTER 


New  Right 
Throttle  Grip 


Precision  Engagement  is  the  largest  upgrade  in  the  history  of  A-10 


■  Significantly  upgraded  and  changed  the  platform,  providing  an  integrated 
avionics  suite  with  a  considerable  number  of  functions  moved  into  software 


■  New  aircraft  baseline  provides  a  point  of  departure  for  many  new  operational 
and  sustainment  capabilities 

KIHOMAC  - 


System  Acquisition  Excellence 


A-10  2030 


“To  Infinity  and  Beyond. . .  ” 


KIHOMAC 


fc  Future  programs,  post-PE,  will  be  forced  to  be 


smaller,  generally  sustainment-bas'eVy: 
a  focus  on  form/fit/function  replacemei 


rograms  with 


Enterprise  architecture  maximizes  the  bang  for  every 
dollar  spent 

V  /  _  A 


CENTER 


System  Acquisition  Excellence 


Avionics  Sustainment  Program  (ASP) 

(Wish  List) 


OGDEN  AIR  LOGISTICS  CENTER 


Replace  w/  MFCD 


Flight  Instruments 
D,  R,  M,  O,  P,  I 


Engine  Instruments 
D,  R,  M,  O,  I 


ALR-69/ALQ-21 3  (addressed  in  EWSU) 

v\/ 

Fuel  Quantity  ID 

D,  R,  M,  0,  P,  G,  1 

- 1 

- - - D,  R,  M,  0,  P,  G,  F,  1 

\ 

U  \m  lavN 

Replace  w/ software 

Heading  &  Attitude  Reference  System 
n  p  iwi  n.  p,  g,  F,  I 


Replace  w/  COTS  B/U  instrument 


Integrate  with  OFP 


1  Communication/Navigation 
J - -  R.  M,  G,  I 


Nav  Mode  Select  Panel 
D,  R,  M,  O,  P,  G,  I 


Replace  w/ software 


I 


- raQj  Caution  Annunciator  Panel 

3^  D,  R,  M,  O,  P,  G,  F,  I 


Replace  w/  software 


Intercom  Panel 
D,  R,  M,  O,  P,  G,  F,  I 


Replace  w/ software 


Control  Display  Unit 
D,  R,  M,  O,  P,  G,  F 


T 


Replace 


Data  Transfer  System 
D,  O,  P,  G,  F 


Upgrade/Consolidate 


Digital  Video  and  Data  Recorder 
D,  R,  M,  O,  P,  G,  F,  I 


Key 

D  -  Diminishing  Manufacturing  Sources  & 
Material  Shortages  Issues 
R  -  Reliability  Issues 
M  -  Maintainability  Issues 
O-  Obsolescence  issues 
P  -  Performance  Issues  (Operations) 

G  -  Growth  Issues 

F-  Function  is  duplicated  elsewhere 

I  -  Not  integrated 


Upgrade/Consolidate 


Out  of  Cockpit  systems 

Stall  Warning  System  -  D,  R,  M,  O,  P,  G,  F 

Embedded  GPS/INS  -  R,  M,  P,  G 

Data  Distribution  -  none 

TEMS/ADR-P,  G,  F 

Radar  Altimeter  -  R,  M,  P,  G 


Replace  w/  software 
Update 

Develop/Integrate 

Upgrade/Integrate 

Replace 
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System  Acquisition  Excellence 


A-10AVSATA  Vision 


OGDEN  A  JR  LOGTSTTCS  CENTER 


a 


A-10  Integrated  Lifecycle  Management  Process 
A-10  Weapon  System  Roadmap 


AVSATA  FY07  -  FY12-13:  Avionics  Architecture/Roadmap 


Analysis  »>  Multiple  OAs  »\  Permanent  Mods  »>  ASP 


3400  POM 
594,  592,  583,  540,  MSD 


PEM  POM 
30XX  3600 


Sustainment  and  Modernization  Modifications 

SLEP,  Wing  Replacement,  PE,  Suite  Updates,  Consolidated  Mod,  etc 


KIHOMAC 


System  Acquisition  Excellence 


A-10  Integrated  Architecture 

OGDEN  AIR  LOGISTICS  CENTER 

■  AVSATA  provides  the  framework  to  help  make  the  most  of  the 
resource  limited  sustainment  programs 

■  Integrated  architecture  provides  a  comprehensive  plan  for  the 
operational  and  technical  capabilities  and  interconnections 
required  by  the  aircraft  lifecycle  sustainment 

■  Defines  a  roadmap  to  show  smaller  programs  how  they  can  fit  into 
the  overall  plan 

■  Defines  a  way  to  leverage  small  sustainment  investments  into 
significant  increased  platform  sustainment  and  capability 

■  Path  Finding  process  applied  to  legacy  sustainment 

■  Keep  A-1 0  relevant  in  our  nations  conflict  and  at  the  forefront  of  the 
force  throughout  its  lifecycle. 


KIHOMAC 


System  Acquisition  Excellence 


OGDEN  AIR  LOGISTICS  CENTER 


ARCHITECTURE  & 
REQUIREMENTS  OVERVIEW 


KIHOMAC 


System  Acquisition  Excellence 


Integrated  Architecture 
Overview 


J OGDEN  AIR  LOGISTICS  CENTER 

■  What  is  an  architecture? 

■  “The  structure  of  components,  their  relationships,  and  the  principles  and 
guidelines  governing  their  design  and  evolution  over  time”  -  DoD  Integrated 
Architecture  Panel 

■  What  is  an  “integrated”  architecture? 

■  Architecture  is  an  integrated  architecture  when  products  and  their  constituent 
architecture  data  elements  are  developed  such  that  architecture  data 
elements  defined  in  one  view  are  the  same  (i.e.,  same  names,  definitions,  and 
values)  as  architecture  data  elements  referenced  in  another  view. 

■  What  are  the  advantages  of  integrated  architectures? 

■  Facilitate  an  organized  and  consistent  standardized  design  process 

■  Facilitate  the  clear  definition  and  implementation  of  new  operational,  system 
&  technical  requirements 

■  Promote  interoperability 

■  Required  by  Joint  Capabilities  Integration  &  Development  System  (JCIDS)! 

■  Provide  for  traceability  of  system  requirements  back  to  the  originating  joint 
concepts  (facilitates  successful  POM  inputs,  i.e.,  getting  program  funding) 

■  Facilitate  systems  and  systems  sustainment  engineering 

KIHOMAC  — - - - - 


System  Acquisition  Excellence 


Fundamental  Linkages 
Between  Views 


OGDEN  AIR  LOGISTICS  CENTER 


KIHOMAC 


System  Acquisition  Excellence 


Traceability  of  Requirements 

from  Users  to  Acquirers  to  Contractors 


A- 10  System 
Program  Office 


System  Functional 
Architecture 


Technical 

Requirements 


OGDEN  AIR  LOGISTICS  CENTER 


ACC/AN  G/AFRC 


Operational 

Requirements 


Operational 

Architecture 


Design 

Requirements 
&  Technical  Specs 


System  Physical 
Architecture 


A-10  TLPS 
Contractors 


KIHOMAC 


Requirements  are  tightly  coupled  to  Architectures 


System  Acquisition  Excellence 


OGDEN  AIR  LOGISTICS  CENTER 


A-10  ARCHITECTURE 
DEVELOPMENT 


KIHOMAC 


System  Acquisition  Excellence 


A-10  Architecture 
and  External  Docs 


OGDEN  AIR  LOGISTICS  CENTER 


A-10  Connectivity  CDD 


■»  CPD’s 


*  MA  ICD’s  were  directed  to  be  converted  or  they  were  rescinded  effective  June  2008 
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System  Acquisition  Excellence 


Architecture 

-Top-Level  View 
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KIHOMAC 


System  Acquisition  Excellence 


Operational  Architecture 


OGDEN  AIR  LOGISTICS  CENTER 


KIHOMAC 


System  Acquisition  Excellence 


4  k 

System  Architecture 

2EZV  OGDEN  AIR  LOGISTICS  CENTER 


KIHOMAC 


System  Acquisition  Excellence 


Architecture  Layering 


OGDEN  AIR  LOGISTICS  CENTER 


KIHOMAC 


System  Acquisition  Excellence 


EXAMPLE 

From  JCAS  CON  OPS: 

-  Establish  &  Maintain  Battlespace  Awareness 


KIHOMAC 


System  Acquisition  Excellence 


JCAS  CO  NO  PS  in 


DOORS 


OGDEN  AIR  LOGISTICS  CENTER 


Joint  level  documents  are  imported 
into  DOORS  and  any  data  deemed 
operationally  significant  to  the  A-10 
is  marked  for  inclusion  into  the 
Operational  Architecture. 


i  EE  7  &  £1 


©■■5.4  D.  Combat  Identifies 


T 


lie]  -  DOORS 


:r  Help 

e?jj 


d  joint  CAS 


g  Q 

LapauiiiLy  'J 1 111  lequiie  uie  auiiiLy  lu  jllcpl 

timing,  positron  and  control  information  from 
a  common  external  source  {Threshold  and 
Objective). 

CAS  procedures  need  to  be  standardized  so 
that  any  segment  of  the  joint  force  can 
effe  ctive  ly  use  th  is  j  o  in t  fire .  Jo  int  d  o  ctrin  e 
needs  to  be  specific.  Accompanying  Service 
doctrine  must  support  and  expand  upon 
these  joint  procedures  {Threshold). 


0-  5.5  E.  Requirements 
El  5.5.1  C4I  Integration 
•  (U)  The  overaref 


Decision  Analysis  Tool  (COA 
analysis) 


Commanders  and  mission  planners  require 
COA  tools  to  determine  the  best  way  to 
prosecute  the  attack  {Threshold). 


••••  (U)  All  proposed 
••••  (U)  The  joint  sen 
|  El  (U)  Table  V-1  list 
L  -  »  Table 

|  [+]••  5.5.1 .1  a.  Collab 

0  - 5.5.1 .2  b.  Autom 
|  0  5.5.1 .3  c.  C4I  S> 

|  S- 5.5. 1.4  d.  Multipl 
0  - 5.5.1 .5  e.  Fused 
0-  5.5.1 .6  f.  Integra 
©  5.5.1 .7  g.  Weap 
0  - 5.5.1  .S  h.  Stand; 
0-  5.5.1 .9  i.  Decisic 
0- 5.5.1  10j.  Redui 
|  H  5.5.1 ,11k.  Near 
0- 5.5.1 .121.  Cornu 
0  - 5.5.1 .13  m.  Stan 
0-  5.5.1 .14  n.  Rear 
ED  -  5.5.1 .15  o.  Oper 
0-  5.5.1 .16  p.  Comr 
El  5.5.2  Planning 
El  -  5.5.3  Preparation 
El  5.5.4  Execution 
El  -  5.5.5  Training 


Redundant,  interoperable, 
filterable,  seamless  systems 
(voice,  data,  text,  graphic, 
imagery;  GIG-enabled) 


Near  real  time/real  time 
battlespace  awareness  (BA) 


X 


Common  C4ISR  architecture 


X 


Standardized,  Digitally 
Produced  Database 
repository  for  all  echelons 


X 


Systems  must  be  reliable,  effective  and 
interoperable  in  any  data  source  for  mission 
success.  An  information  exchange  must  be 
independently  filterable  at  each  interface 
[(Threshold  and  Objective). 

Data  flow  must  be  continuous  and  constantly 
updated  (refresh  data  <10  seconds)  with 
the  latest  information  so  as  to  ensure  data 
latency  is  not  a  factor  in  the  decision  making 
process.  The  system  can  prioritize  its 
information  exchange  independently  at  each 
interface  (Threshold  and  Objective). 

The  CAS  operational  architecture  shall 
provide  an  internal  growth  capability  through 
an  open  C4ISR  systems  architecture 
approach;  flexible  for  introduction  of 
improved  capabilities  via  technical,  tactical  or 
I  procedural  improvements;  having  a  web- 
based  look  and  feel  for  ease  of  use  and 
!  Ifamiliarity  (Threshold). 

The  repository  will  afford  the  user  stored  and 
near  real  time  informat  ton  from  a  number  of 
so u rces.  Rep osito ries  at  d iff e re nt  echelons 
will  be  able  to  "push/ pull"  data  as  the  user 
!  dictates  {Threshold  and  Objective). 


0--5.6F.  Information  Exchc 
EJ  -5.7G.  Key  Performance 
El-  6  Appendix  A.  Operational  C 
EE  71.  Introduction 


Reachback  capability 


Sufficient  bandwidth  and  relay  capabilities  will 
be  available  for  joint  force  and  components 
to  coordinate  and  integrate  with  agencies 
outside  the  theater  of  operations 

(Threshold!. 
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-  Establish  &  Maintain  Battles  pace  Awareness 
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Operational  Architecture  Thread 

-  Establish  &  Maintain  Battlespace  Awareness 
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Operational  Requirements 

-  Derived  from  Operational  Architecture 
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A  10  Documents  (Formal  module)  -  DOORS 


1  File 

Edit  View  Insert  Link  Analysis  Table  Tools  User  Help 

1:  y  &  0  \  mr  m  m*  j  §“  g*3  J  3*  H3  ^ 

:  ©  ©jj!  ©5  ©? 

1 :  View  Standard  view  v  All  levels  v  ^ 

.2  i  IS  7  , 

F  ii  

Fi 

■>  1  1 

5.3. 
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Networked, 
fast,  and 
lethal 


Networked, 

Precise, 

Adaptable/ 

Tailorable, 

and 

Ixpeditionary 


Mach  ine-to-Mach  ine 
(M2M)  Information 
3rocessing. 


Digital  Map  and  Near  Real- 
Time  Imagery. 


transmitted  by  the  transmitting  node  and  its  coordinates  ar 

adjusted/altered/rounded  beyond  one  one-thousandths  (.01 
I  minute 


A-10  documents  such  as  the  Connectivity 
CDD  are  linked  to  the  Operational  Views 
and  specific  system  requirements 


A/OA-10  Connectivity  shall  provide  M2M  targeting  with  no  manual 
entry  or  reentry  and  with  the  accuracy  needed  for  sensor  and 
weapon  cueing 


A/OA-10  Connectivity  shall  provide  capability  for  storage  and  display 
of  existing  digital  maps  at  scales  of  1:1M  and  1:500K  for  an  area  of 
170,000  square  miles  (approximate  size  of  Iraq  or  comparable  sized 
area  in  any  theater  of  operation);  or  1:250K  for  an  area  of  85,000 
square  miles  (approximately  half  the  size  of  Iraq  or  comparable 
sized  area  in  any  theater  of  operation) 


jsame  as  Threshold 


I  The  A/OA-10  Connectivity  modification  should  provide 
capability  for  storage  and  display  of  existing  digital  maps  at 
I  scales  of  1:50K,  1:100K,  1:250K,  1:500K,  1:1M,  1:2M,  and 
1:5M  as  moving  and  static  maps  for  an  area  of  170,000 
j  square  miles 


Operational  necessity 
to  reduce  pilot 
workload  (HSI/HFE) 
and  for  the  most 
effective  and  efficient 
weapons  employment, 
and  minimize  F2T2EA 
kill  chain  times.  Sensor 
cueing  may  also  help 
ore  vent  fratricide  when 
linked  to  the  BFT 
information  available 
on  the  tactical  net. 


oy  the  pilot  with  a  high 
confidence  level 


Values  calculated  to 
enable  maximum 
storage  within  existing 
mass  memory  capacity 
and  required  area 
oased  on  OIF  and  OEF 
lessons  learned.  Intent 
is  to  keep  program 
costs  down  by  not 
'equiring  new  memory 
capacity  for  this 
modification,  while  still 
meeting  operational 
'equipments. 
Operational  necessity 
to  provide  pilot 
situational  awareness 
in  intended 

environment  (HSI/HFE). 


||  14.1  Key  System  Attributes. 

(Diagnostics  and  Prognostics.  Collection,  refinement,  compilation,  integration,  and  distribution  of  aircraft  status  elements  into  an  overall  support  database  are  critical  to  successful  A/OA-10  employment  throughout  its  entire  operational 
spectrum.  A/OA-10  Connectivity  will  integrate  applicable  and  effective  on-board  monitoring/ recording  devices  and  software,  i.e.,  Built-In-Test  (BIT),  that  provide  enhanced  capability  for  fault  detection  and  isolation,  monitor  various 
components  and  indicate  out  of  range  conditions,  imminent  failure  probability,  and  similar  proactive  maintenance  optimization  actions  to  optimize  the  time  to  repair  of  the  Processor.  Emphasis  must  also  be  on  accuracy  and 
minimization  of  false  alarms.  On-board  collected  data  shall  be  provided  so  it  facilitates  the  immediate  start  of  qround  servicing  and  other  recovery  actions  as  well  as  a  more  detailed  and  comprehensive  analysis  of  in-flight  failures  and 
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System  Requirements 

-  Derived  from  Operational  Requirements 
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Si  ’SADLJEPLRS_Example'  current  0.0  in  /  A-10  System  Requirements/A-10  Baseline  Requirements  (Formal  module)  -  DOORS 

File  Edit  View  Insert  Link  Analysis  Table  Tools  User  Help 

j  y  ®  DJ  j  m  a*  a*  j  f  f  J1  a*  N5  fr 

j  ©-:  S*  ©j^ 

s? 

j  View  ^Public  v  All  levels  v  £  , 

if  iTT  (5k  a 

i  7  #  A  tl 

F 

ID  g  Function 

Subfunction 

Threshold  Requirement 

General 

Interoperability 

Platforms 


The  A- 10  shall  be  capable  of  providing  digital  data  and  voice  co 
the  Tactical  Air  Control  System  (TACS)  that  include,  but  not  be 
platforms: 

-A- 10 
-  F-16 


The  A-10  system  requirements  are 
stored  in  DOORS  and  are  traced  to  the 
Operational  Views,  System  Views,  as 
well  as  documents  such  as  the 
Connectivity  CDD  and  PIDS. 


120  COMM-004 


General 
Data  Link 


-  Joint  Terminal  Attack  Controllers  (JTAC) 

-  Tactical  Air  Control  Parties  (TACPs) 

-Air  Support  Operation  Center 

-  Air  Operations  Center, 

-  Forward  Air  Controller  Airborne  (FAC (A))  aircraft 

-  Command  and  Control  (C2)  aircraft  (e.g.,  E-3  AWACS,  E-8  JSTARS,). 


The  A-10  shall  support  a  selected  DoD  standard  digital  data  link  communication  capability. 


-  UK  Tornado 

-  UK  Harrier 


Same  as  Threshold. 


121 


ICOMM-005 


4  Comm  General 
Data  Link 
Interoperability 


m\  < 

Username:  Ryan  Exclusive  edit  mode 


The  A-10  data  link  system  must  folly  support  execution  of  joint  critical  operational  activities 
identified  in  the  applicable  joint  and  system  integrated  architectures  and  the  system  must 
satisfy  the  technical  requirements  for  transition  to  Net-Centric  military  operations  to  include: 

1)  DISR  mandated  GIG  IT  standards  and  profiles  identified  in  the  A/O A- 10  Technical  Standards 
View  (TV-1), 

2)  DISR  mandated  GIG  KIPs  identified  in  the  Key  Interface  Profiles  (KIP)  declaration  table, 

3)  Net  Centric  Operations  and  Warfare  Reference  Model  (NCOW-RM)  Enterprise  Services 

4)  Information  assurance  requirements  including  availability,  integrity,  authentication, 
confidentiality,  and  nonrepudiation,  and  issuance  of  an  Interim  Approval  to  Operate  (IATO)  by 
the  Designated  Approval  Authority  (DA A) 

5)  Operationally  effective  information  exchanges;  and  mission  critical  performance  and 
information  assurance  attributes,  data  correctness,  data  availability,  and  consistent  data 
processing  specified  in  the  applicable  joint  and  system  integrated  architecture  views. 


The  A-10  data  link  system  should  folly  support  execution  of  all 
operational  activities  identified  in  the  applicable  joint  and 
system  integrated  architectures  and  the  system  must  satisfy  the 
technical  requirements  for  Net-Centric  military  operations  to 
include: 

1)  DISR  mandated  GIG  IT  standards  and  profiles  identified  in 
the  TV-1, 

2)  DISR  mandated  GIG  KIPs  identified  in  the  KIP  declaration 
table, 

3)  NCOW  RM  Enterprise  Services 

4)  Information  assurance  requirements  including  availability, 
integrity,  authentication,  confidentiality,  and  nonrepudiation, 
and  issuance  of  an  Interim  Approval  to  Operate  (IATO)  by  the 
Designated  Approval  Authority  (DA A) 
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EPLRS  SADL  Display  Data 


EPLRS  SADL  Control  and  Data  Entry  Signals 


4.2 


COMM  MX  Data  Upload 

SOI  SPI  State 

A1 

TAD State 

Weapon  Inventory 

ENAV  Aircraft  State 

COMM Mission Data Upload 

V. 

Fuel System Display Data 

EPLRS  SADL  RF  Ext, 


4.9 

EPLRS  SADL  RF  Interface 


EPLRS_SADL_RF 

V  43 


COMM  MX  Data  Upload  x 

EPLRS SADL Crypto  ^ 


EPLRS  SADL  Network  Manager 


EPLRS  SADL  Configuration 


Host_Format_EPLRS_SADL_Message<; 


E  P  LR  S_S  AD  L_B  IT 
4.6 


EPLRS  SADL  Status  BIT 


EPLRS  SADL  BIT 


EPLRS  SADL  Manager 
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The  functions  are  traced  from  the 
system  requirements  which  they  fulfill 
as  well  as  any  associated  Operational 
Views. 


System  Physical  Architecture 

-  Implements  System  Functional  Architecture 
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Roadmap  Process 
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Baseline 

Evaluation  &  Trade  > 

To-Be 

Architecture 

Studies 

Architecture 

^ _ _ _ / 

Sustainment  & 
Modification 
Programs 
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Notional  SV-8 


Systems/Services  Evolution 


A-10  Weapons  CDD 


MU  OS  SATCOM 
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Notional  SV-9 
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Technology  Forecast 
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Key 


Technical  Maturity  Level 


Low  High 
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W  A  VS  A  TA  Results 

W 

OGDEN  AIR  LOGISTICS  CENTER 

■  AVSATA  already  resulted  in  integrated  system  on  A-10 

■  Distributed  mass  memory  (greater  map  and  data  storage), 

■  Helmet  mounted  cueing, 

■  LARS  VI 2,  Integrated  personnel  recovery  systems  for  use 
during  CSAR, 

■  Expanded  bus  infrastructure  to  support  future  high  speed 
devices  (12  Port  1GB  Ethernet  switch) 
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Tying  Requirements  to 
Funding  Requests 

(U)  A-  lOAvionics  Sustainment  Program  (ASP) 


OGDEN  AIR  LOGISTICS  CENTER 


BACKGROUND : 


(U)  A- 10  avionics  system  has  aging  Line  Replaceable  Units  (LRU)  that  have  exceeded  their  design  lives.  The 
reliability,  sustainment,  and  obsolescence  issues  are  decreasing  aircraft  availability,  increasing  maintenance  costs, 
and  limiting  growth,  mission  readiness  and  capability. 

ADJUSTMENT: 

'  '  -  —  -  -  -  -  '  ’  "  - - Jsj 


(U)  Develop,  procure,  and  install  344  ASP  kits  on  A-10  fleet  (replace  26  aging  LRUsJ. 
high  maintenance  drivers  with  obsolescence  issues  with  5  new  LRUs  pep 


displays, 


$M:  XXXXXXXXXX 

ADJUSTMENT 
REV  PGM  TOTAL 


FY10 


FY11  FY12 


FY13 


MPWR 

mi 

R14 

m?  me  ra? 

OFF 

0 

0 

0 

0  0  0 

ENL 

0 

0 

0 

0  0  0 

CIV 

0 

0 

0 

0  0  0 

(U)  RQMTi^CQRD,  Dec  04;  A-10  EW  CDD,  A-10  Connectivity  CDD,  Mar  0  7 ;  JCAS  MA  ICIlt  Jun  04,  JCAS 
CONOPS ;  MOOS,  Joint  Fires  ICD,  Nov  02,  GIG  MA  Nov  02 

-  (U)  If  not  funded,  Continued  decrease  in  Aircraft  Availability 

-  (U)  If  not  funded,  Grounding  of  aircraft  due  to  nnsuppor table  CDU,  HARS,  and  displays  in  FY14-FY17 
(U)  If  not  funded,  Mission  non  capable  -no  (processing)  growth  path  as  processors  are  maxed  out.  FY17 
(U)  If  not  funded,  Projections  lead  to  capability  gaps,  FY17  through  FY2S 
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Engineering 
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Richard  L.  Sorensen  is  a  Staff  Systems  Engineer  at  KIHOMAC  Inc.  He  has 
over  twenty  eight  years  experience  in  systems  engineering  and  systems 
architecture  in  both  military  and  civil  applications. 

Adam  Grimm  is  Director  for  Strategic  Programs  at  KIHOMAC  Inc.  He  has  over 
eight  years  working  logistics,  engineering  and  requirements  for  U.S.  Air 
Force  aircraft  and  net-centric  and  command  and  control  systems. 

Jerry  L.  Coates,  M.  E.  E.  E.,  is  the  A-1 0  OSS&E  Integrator  for  the  A-1 0  System 
Program  Office  (OO-ALC/  538th  ACSG/EN).  He  has  21  years  of  experience 
with  the  USAF  at  OO-ALC  including  2  years  as  an  AF  Exchange  Engineer  in 
Manching,  Germany  at  the  German  Airworthiness  Certification  Airbase  WTD 
61,  and  11  years  of  experience  in  industry  (Boeing,  SSAI,  Robert  Bosch  and 
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KIHOMAC 


System  Acquisition  Excellence 


Da/nl'jfjm-inl 

Ocivbnr'^2 


SmallSat  Conceptual  Desiynm 
Trade  and  Cost  Modeling  Tool 


Dr.  Deganit  Armon 


NDIA  12 
Engined 


Ting 
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0  Team 


•  Advatech  Pacific,  Inc.  • 

■  John  Carsten 

■  Deganit  Armon 

■  Dana  Sherrell 

■  Michael  Paisner 

■  Mark  Sutton  • 

■  Steve  Mysko 

■  Paul  Dorman 

■  David  Cantwell 

•  MCR  LLC  • 

■  Roy  Smoker 

■  Daniel  Feldman 


Air  Force  Research  Laboratory, 
Space  Vehicles  Directorate 

■  Brent  Hamilton 

■  Ross  Wainwright 

Tecolote  Research 

■  Al  Milton 

■  John  Trevillion 

■  Darren  Elliott 

Rocket  Science  Solutions,  Inc. 

■  Jerry  Sellers 
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Advatech  Company  Overview 


•  Founded  in  1995,  Owned/Operated  by  Aerospace  Engineers 

•  Locations  in  California,  Arizona  and  Virginia 

•  R&D  and  Engineering  Services 

■  Integrated  tool  development  and  analysis 

-  Space  vehicle  modeling 

-  Launch  vehicle  design  and  cost 

-  Hypersonic  vehicles 

-  Trajectory  analysis 

-  Range  safety  (Responsive  Range  Safety) 

■  Software  design  and  development 

■  Engineering  design  and  analysis 

-  Structural 

-  Thermal 

-  Composites 

-  Advanced  space  propulsion  (electric  /  nuclear) 

-  Tactical  communications 
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T raditional'Design  Approach 


Early  Design  Challenges  of  High-Performance  Compi 

hat  is  the  current  design  process? 


Syst€ 

Requiremefl 

■  Mission  requirements 

■  Size,  weight,  power 

■  New  technology 

insertion 

■  Performance 

■  Schedule 


Integrated  Design  Approach 


Solution  to  Early  Design  Challenges  of  High-Performance  Comple: 

optimizing  tools  provide: 


System 

Requirements  I 

1 

■  Mission  requirements  \ 

■  Size,  weight,  power  / 

■  New  technology  ( 


Integrated*  T  oollSu  ite 


Continuous  trade  study  capability  throughout  the 
£ycle 

model  key  parameters  m  the  pre-system  acquisition 


■  Co 
■Sc 

■  Technology  risk 

Responsive  tumarouna'-  djfejlP^eeks! 
System  and  subsystem  tradfiHJfolygj 
Continuous  knowled 


Fundament 
withi 


ul  enabler  for  building  ihe  beet 
h  die  cost,  schedule  and  mill m 


rforming  system 
ogy  constraints 


Advatech  Integrated  Projects 

•  Space  vehicle  design  and  cost  (ACES-ISET) 

•  Advanced  Cost  Model  (ACM) 

•  Launch  vehicle  design,  operations  and  cost  (IPAT) 

•  Hypersonic  aeromechanics  tool  (IHAT,  FPAT) 

•  Integrated  Physics  Based  Cost  /  Risk  Analysis  Tool  (ICAT) 

•  Composite  Rotor  Blade  and  Wing  Structural  Design  Tool 

•  Component  Integrated  Modeling  Simulation  and  Test  Analysis  Environment 
(CIMSTA) 

•  Naval  Engineering  Analysis  Tool  (NEAT) 

•  Virtual  Satellite  Integration  Effort  (VSIE) 

•  Small  Satellite  Launch  Vehicle  (SPRITE) 

•  Analytical  Methods  for  Sandwich  Core  Termination 

•  Integrated  High  Payoff  Rocket  Propulsion  Technology  (IHPRPT) 

•  Aircraft  Vulnerability  Model  (AVM) 

•  Combined  Hall  Effect  Thruster  Code  (CHETC) 

•  Field  Reverse  Configuration  (FRC)  Thruster  Model  -  Orbit  Transfer  Vehicle 
System  Model 

•  Highly  Mobile  Tactical  Communications  (HMTC) 

•  Integrated  Solid  Motor  Analysis  Tool  (ISMAT) 


SmallSat 


Tool 


Advanced  Computational  Engineering  Simulator  - 
Integrated  Space  Analysis  Tool  (ACES-ISET) 

Customer:  Air  Force  Research  Laboratory,  Space 
Vehicles  Directorate,  Kirtland  AFB,  NM 


Partners:  Tecolote  Research,  MCR  LLC,  RSSI 


•  An  integrated,  multi-disciplinary  engineering  tool  suite 

■  Optimizes  the  design  and  cost  of  space  vehicles 

■  Models  the  space  environment 

■  Selection  of  launch  vehicles  and  modeling  launch  operations 

■  Perform  mission  planning  trade  studies 

■  Visualization  of  results 


Integrated  System  and  Cost 
Model  (ISCM)  -  Tool  Suite 


Space  Vehicle 
Module 

Space  Vehicle  Design  SMAD) 
Space  Vehicle  Propulsion 
Orbit  Propagation 
Space  Vehicle  Costing  (ACEIT) 

•  Radiation  Exposure 

•  Radiation  Detector  Response 

•  On  Orbit  Operations 


Historical/Knowledge  Database 


P  ACES  ConcSptoaQDe 


Cost  Estimating 


•  Cost  modules  fully  integrated  with  design  tools 

•  Cost  estimating  relationships  based  on 

■  historical  data 

■  sub-system  weights 

■  materials 

•  Historical  data  used  to  identify  cost  growth  rates 
related  to  Technical  Readiness  Levels  (TRL) 

•  Cost  and  schedule  are  related  to  TRL  and 
system  engineering  milestones 

•  Built  in  risk  estimating  capabilities 


^  Cost  Risk  Estimates 

Cost  growth  incurred  as  technology  matures 


'I  HL.  5-6 


FCA  - 

TBL  6-7 


IOC 

TRL  7-8 


TriangulanJftiiflds 
(L,M,H)  on  weights 
HWfeS-Curves 

Determined  using 


S-Curves  shil 


deterministic  risk 
analysis  tool 
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Applications 


•  Examples  of  Trade  Studies 

■  Effect  of  subsystem  reduction  on  total  vehicle 
design 

■  Concept  evaluations  of  proposed  TacSat-5 
concepts 

■  Cost  impact  of  alternative  TacSat-3  designs 

■  Launch  vehicle  selection  and  cost  for  satellite 
constellation 

■  Trajectory  analysis  for  DSX  alternative  orbits 

■  Concept  modeling  for  ORS  modular  satellite 
architectures 
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Sensitivity  Study 


Study  highlights 

tefiyantitative  and  qualitative  data  on  the  impact  of  decreasing 
Ijfca  Weightiand  Power  (SWAP)  of  individual  subsystems  on 
BflBHfe  ra  1 1  ffefieweh  i  cl  e  SWAP 


■01 

fact 


ra  vehicle  subsystems  &  components  interaction 
asjbility  of  reducing  Space'^febicle  mass  by 


'Identified  two  major  are; 

-  energy  conversion 

-  structural  material^H 

Presented  at  the  6th  ReJ! 


rstem 


future  fo< 


research 


h  2008 


Expectation|lf< 

quantifying 


tan  be  bound  by 
ireakthrough 
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Sensitivity  Study  -  Sample  Results 


100% 


50% 


Integrated  DcRitjmApgj 


Solution  to  Early  Design  Challenges  of  High-Performance  Complex  Systems 

mg  tools  provide: 


Inibyrxib 


gr)  TacSat-5  Concept  Evaluations 


Study  highlights 

^Source  solicitation  evaluation 

HjH|^i^^ffl||^lBfril|^iclassified  and  unclassified) 

■  Identified  and  quantified  issues  with  concept  proposals: 
jn  Projected  costs  that  exceed  available  budget 

|  Costs  that  assume  payload  desiglNjeritages  not  suppoj|ed  in  proposal 

-  HicW^ravlo^^BBTOtewTR^PBWhidevelopmenftbhedules  or 
payloads  exceeding  mass  budget  limits 

■  Identified  aiWIquantifilBl^iriPwith  tran 
version 

■  Demonstrated  that|  sord^OTposalsfiS^ 

-  Contained  inconsist^^^^mptWflWt^^^ 

-  Contained  questiprf^Wgj^^MlTflMn^nS^hg^itfflPecI  further  investigation 


to  operaiRjffel 


Knowledge? 
decisions  abc 


Ign  phase  enabled 
ilities  before  a  large 


mitted 


Alternative  TacSat-3  Designs 


Study  highlights 

.Ongoing  study 


eBtlba/fitem  design  changes  needed  to  create  an 
ionalized”  version  of  TacSat-3. 

luate  design  modiTifcatjons 


-  Increased  missi 

-  SubsyS^^reaeglBnWIOTUBwer  tech 
Determine  cost  of  de 


Select  and  deterge 
vehicles  needed 


edification 


rie  procurerrier 
o  launch  a  satellite  coris 


nch 

ion 


Cost  estiimjg 
subs 


s  for  design  modification 
ysiern  heritage  arid  techrji 


5  are  affected  by 
•logy  maturity. 


Lessons  Learned 


•  Selecting  integration  environment 

■  License  cost 

■  Performance  (speed) 

■  Portability  (platforms) 

■  Flexibility  and  ease  of  development 

■  Scalability 

■  Automated  parameter  management  (facilitates  trade  studies) 

■  User  interface 

•  Selecting  M&S  tools  to  be  integrated 

■  Existing  customer  tools 

■  Validation  level  (industry  accepted) 

■  OTS  versus  development 

•  Data  availability  and  reliability 

■  Proprietary  data 

■  Validation  level 

•  Export  control  and  use  restrictions 

•  Managing  customer  expectations 
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M)  Conclusio 


Integrated  tools  suites 

•►^prayide^ubstantiated,  traceable  and 
lbng£|o8fotfble 

™  reveal  interdependencies  of  cost,  risk, 
scheomSkand  performance 


pTovr 

schei 


igher  con 
(estimate: 


ice  n 


enable  better  manan 
investment  by  deck 


lent  of  tech i 
makers 


Conceal] 
to  det 


Billable 

pace 


JOHNS  HOPKINS 

UNIVERS  ITY 


An  Integrated  RAM  Approach  to 
System  Design  and  Support 


Bob  Finlayson 

The  Johns  Hopkins  University 
Whiting  School  of  Engineering 


Challenge 


JOHNS  HOPKINS 

UNIVERS  ITY 


“The  operations  and  support  phase  of  the  system  life 
cycle  is  the  time  during  which  the  products  of  the  system 
development  and  production  phases  perform  the 
operational  functions  for  which  they  were  designed.  In 
theory,  the  tasks  of  systems  engineering  have  been 
completed.  In  practice,  however,  the  operation  of 
modem  complex  systems  is  never  without  incident.  ” 


Kossiakoff,  A.,  Sweet,  W.,  Systems  Engineering  Principles  and  Practice 


Reliability,  Availability,  Maintainability 


JOHNS  HOPKINS 

UNIVERS  ITY 


•  The  most  important  aspect  of  O&S  design 

•  Directly  influences: 

-  Operational  effectiveness 

-  Safety 

-  Supply  support 

-  Maintenance  planning 

-  Manpower  and  personnel 

-  Cost 

•  Indirectly  influences: 

-  Management  of  the  Supply  Chain 

-  Technical  Data 

-  Facilities 

-  Support  Equipment 


JOHNS  HOPKINS 

UNIVERS  ITY 


Failure  Due  to  a  Bad  Design 


Tacoma  Narrows  Bridge 
07  November  1940 


Tacoma  Narrows  Bridge 
2006 


Failure  Due  to  Multiple  Factors 


JOHNS  HOPKINS 

UNIVERS  ITY 


Interstate  35  Bridge,  Minneapolis,  MN,  01  August  2007 
-Excessive  loads  -  using  the  bridge  in  an  unintended  environment 
-Undersized  gusset  plates  -  poor  design 
-Corrosion  -  improper  maintenance 
-Design  -  geometric  failure  (bowing  of  the  U1 0  joint) 


Program  Failure  Waiting  to  Happen 


JOHNS  HOPKINS 

UNIVERS  ITY 


•  “The  reliability  for  the  ABC  system  shall  be  0.97  and  it  is 
listed  in  the  Capabilities  Development  Document  as  a 
Key  Performance  Parameter.” 

•  “How  did  you  arrive  at  0.97?  Was  it  derived 
mathematical  based  on  similar  systems,  what  current 
technology  can  allow,  or  on  requirements  for  mission 
success?” 

•  “None  of  the  above  -  0.97  is  what  the  user  thinks  he  can 
live  with.” 


Calculating  Reliability 


JOHNS  HOPKINS 

UNIVERS  ITY 


Easy 


Synthesis  and 
Design 


Logistics 

Supportability 


Analysis  of 
Alternatives 


What’s  Behind  the  RAM? 


JOHNS  HOPKINS 

UNIVERS  ITY 


Pulling  it  all  together 
ystem  Enginee 


Maintainability 

(Logisticians) 


Structured  Analysis  for  Deployed 
Systems  Engineering 


JOHNS  HOPKINS 

UNIVERS  ITY 


Limitations/Constraints 


JOHNS  HOPKINS 

UNIVERS  ITY 


•  Availability  of  data 

-  Historical,  analogy,  parametric  analysis 

•  Use  of  the  correct  reliability  prediction  model 

-  Do  you  have  a  clear  understanding  of  system? 

•  Understanding/modeling  of  operational  environment 

-  Will  not  account  for  all  of  the  factors  that  affect  system  reliability 

•  Accounting  for  operation  of  the  system  itself 

-  User  may  not  employ  the  system  as  designed 


100%  reliability  does  not  exist  because  of  cost,  acceptability  of 
failure,  and  in  truth. ..it  is  impossible  to  obtain 


Stakeholders/Contributors 


JOHNS  HOPKINS 

UNIVERS  ITY 


•  Similar  systems  in  a  similar  environment 

-  Data  can  help  determine  the  reliability  of  the  new  system 

•  System  designers 

-  Account  for  reliability  throughout  the  design 

•  Evaluators 

-  Assess  if  the  system  meets  reliability  requirements 

•  Manufacturers 

-  Affects  system  assembly  arid  quality  control 

•  Product  introduction  team 

-  Early  barometer  of  system’s  reliability 

•  Product  support  team 

-  Best  assessment  of  reliability  in  system  design 

•  Users  and  maintainers 

-  Ultimately  affects  system  performance;  perhaps  more  than  any  other 
measure 


JOHNS  HOPKINS 

UNIVERS  ITY 


Do’s  and  Don’ts  in  Reliability 
Applications  and  Analysis 


Addressing  Reliability 


JOHNS  HOPKINS 

UNIVERS  ITY 


•  Do  you  truly  understand  the  problem  at  hand? 

-  Environment,  requirements,  CONOPS 

•  Have  you  set  the  boundary  conditions? 

-  Assumptions,  limitations 

•  Have  you  correctly  assumed  equilibrium? 

-  Modeling 

•  Can  you  solve  for  the  unknowns? 

-  Design  and  verification 


•  This  can  be  a  typical  behavioral 
model  for  an  organic  and 
inorganic  system  -  looks  fairly 
benign 

•  In  reality,  randomness, 
environmental  effects, 
catastrophic  events,  etc.,  will 
skew  the  outcome 


0 


Time  (t) 


JOHNS  HOPKINS 

UNIVERS  ITY 


Failure  Due  to  a  Bad  Design 


Tacoma  Narrows  Bridge 
07  November  1940 


Tacoma  Narrows  Bridge 
2006 


Some  Basics 


JOHNS  HOPKINS 

UNIVERS  ITY 


•  Select  the  correct  probability  distribution  function 

-  Exponential 

•  Constant  failure  rate  model  for  continuously  operating  systems 

•  Probability  of  failure  for  some  time  period  in  the  future  is  independent  of  age 
(i.e.,  “memorylessness”) 

-  Normal 

•  Summing  of  random  effects 

•  Assumes  independence  between  events 

•  Events  equally  weighted  (or  no  one  which  is  dominant) 

•  Look  at  component  interactions  and  interfaces 

-  Conditional  probability 

•  Consider  redundancy  in  design 

-  Be  careful! 

•  Don’t  be  afraid  of  Weibull 


Weibull  Distribution 


JOHNS  HOPKINS 

UNIVERS  ITY 


•  Most  widely  used  in  reliability  calculations 

-  Appropriate  use  of  parameters  can  model  a  variety  of  failure  rate 
behaviors  (2  and  3  parameter  distributions) 

•  Used  to  calculate  wear  in  and  wear  out  phenomena  as  well  as 
constant  failure  rates 

-  Take  on  shape  of  other  distribution  functions  by  varying  its  shape 
parameter 


Weibull  Reliability  Plot  w/0< |3<1,  f^l,  (3>1 


0  1 00.00  200HD  3 DO  HD  40DDD  50DD0  6DDD0  7DDHD 

Time,  (£) 


Why  Weibull? 


JOHNS  HOPKINS 

UNIVERSITY 


•  Exponential  assumes  all  units  are  being  utilized  in  exactly  the  same 
manner  and  are  exposed  to  the  same  stresses 

•  However... 

-  Failure  characteristics  vary  depending  on  their  usage 

-  Failure  distributions  are  different  for  different  types  of  systems  (e.g., 
mechanical  versus  electronic) 

•  Normal,  Lognormal  and  Weibull  distributions  are  time  dependent 

-  That  is,  reliability  of  a  device  is  a  strong  function  of  its  age 

-  Require  at  least  two  parameters 

•  Weibull  looks  at  historical  data  or  data  from  a  similar  system  being 
used  in  a  similar  manner  to  make  predictions  on  behavior  and 
failures 


Applying  FMECA 


JOHNS  HOPKINS 

UNIVERS  ITY 


Blanchard,  Benjamin  S.,  “Logistics  Engineering  and  Management  (Sixth  Edition)” 


Failure  Interactions 


JOHNS  HOPKINS 

UNIVERS  ITY 


•  Best  analyzed  through  Markov  Analysis  and  state  diagrams 

•  Calculates  reliability  for  the  following  scenarios 

-  Independent  components 

-  Load-Sharing  systems 

-  Standby  systems 

•  Idealized  system 

•  Failures  in  standby  state 

•  Switching  failures 

•  Primary  system  repair 

-  Multi-component  systems 

-  Combination  of  systems 

•  Availability 

-  Standby  redundancy 

-  Shared  repair  crews 


JOHNS  HOPKINS 

UNIVERS  ITY 


Example 


Systems  Approach  to  Reliability 


JOHNS  HOPKINS 

UNIVERS  ITY 


•  Reliability  can  only  be  increased  if: 

-  Component  reliability  is  increased  through  design  optimization 

-  Component  redundancy  is  built  into  the  system 

-  Interactions  between  the  components  are  understood  to  a  very 

high  degree 


Is  a  single  engine  failure  an  independent  event? 

If  not,  do  you  understand  the  collateral  effects? 

How  do  you  model  them  to  predict  single  engine  failure? 


Summary 


JOHNS  HOPKINS 

UNIVERS  ITY 


•  Requirements 

-  Understand  early  in  system  development  the  reliability  expectations 

-  Determine  if  they  are  achievable 

•  Functional  analysis  and  allocation  and  its  effects  on  reliability 

-  Best  accomplished  through  a  FMECA 

•  Synthesis 

-  Build  reliability  into  the  design  via  component/subcomponent  design, 
minimization  of  component  interactions  and  redundancy 

•  Feedback 

-  Predict  component  and  system  behavior  via  probability  analyses  (e.g.,  Weibull 
distribution  function,  M&S,  etc.) 

-  Determine  if  the  initial  design  will  meet  performance  and  cost  constraints;  iterate 
and  refine  as  required 

•  Verification 

-  Ensure  the  test  and  evaluation  activities  address  those  areas  that  will  provide  the 
greatest  visibility  into  system  reliability;  this  is  often  overlooked 

•  Specification 

-  Hold  the  manufacturer  accountable  for  all  aspects  of  reliability  in  the  design 
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Economics  of  Human  Systems  Integration: 

Early  Life  Cycle  Cost  Estimation  Using  HSI 

Requirements 
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MIT  Graduate  Research  Assistant 
Research  Advisors:  R.  Valerdi  and  D.  H.  Rhodes 

12th  Annual  NDIA  Systems  Engineering  Conference 
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Why  Measure  HSI  Cost? 


Systems  Engineering  Advancement  Research  Initiative 


Aircraft  SE/PM  as  a  Percentage  of  Total  Aircraft  Development  Cost  Minus 
Outlier  Development  Programs,  1960s-1990s 
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Contract  award  date 


RAND  MG4TJ-J.4 


“Systems  Engineering  and  Program  Management  Trends  and  Costs  for 
Aircraft  and  Guided  Weapons  Programs”  -  RAND  Corp. 
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TY  96  $M 


Systems  Engineering  Advancement  Research  Initiative 


HSI  for  Reduction  of 
Total  Ownership  Cost 


□  Procurement  Cost 

□  O&S  Cost 
Development  Cost 


UAS  Performance 


i 


1  X 

UAV  Mishaps 

Aircraft  Mishaps 

Predate  -  32* 

F-ie-a 

Pioneer -3H’ 

General  Aviation  -  1 

Hunter-  55* 

Regional  Commuter  -  0. 1 

“  mucti  tese  than  100,000  flight  hours 

Large  airliners  -  0.01 

i  / ' 

v  Table  3*1  Class  A  Mishap  Rates  Per  100,000  Flight  Hours 
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^5 EjfSr  PM  Concerns  about  HSI  Cost 

Systems  Engineering  Advancement  Research  Initiative 


Major  Recent  UAS 


Averaged 
over  3  years 

Total  Budget 
>$1B,  <$5B 


=  SE/PM  Costs 


Procurement 
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Current  Estimation  Methods 


Systems  Engineering  Advancement  Research  Initiative 


development 

Data  from  Historical 
Systems 


seari.mit.edu  ©  2009  Massachusetts  Institute  of  Technology 


Estimate  of 
SE/PM  as 
Ratio  of  Total 


Current  Estimation  Methods 


Systems  Engineering  Advancement  Research  Initiative 


Data  from  Historical 
Systems 


Factors  influencing 

Estimate 


Expert  opinion 
Technology  Changes 
Aircraft  Weight 
#  Units 


New  Program 

Estimated 
12%  SE 
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'XVparametric  Estimation  of  SE:  COSYSMO 


Systems  Engineering  Advancement  Research  Initiative 
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'XVparametric  Estimation  of  SE:  COSYSMO 


Systems  Engineering  Advancement  Research  Initiative 


Size  Drivers 

#  Requirements 

#  Interfaces 

#  Scenarios 

#  Algorithms 

+ 

3  Volatility  Factors 


Effort  Multipliers 

-  Application  factors 

-8  factors 

-  Team  factors 

-6  factors 

-  Schedule  driver 


Calibration 
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SErVlnputs  Needed  to  Implement  COSYSMO 

Systems  Engineering  Advancement  Research  Initiative  1  1 


Requirements 


-  Counted  from  Requirements  Documents 
(CDD,  ORD) 


-  “shall”  “will”,  “must” 


Requirements  Decomposition 

1 .  Determine  system  of  interest 

2.  Can  requirements  be  test,  verified,  or  designed? 

3.  Sketch  system  of  interest  relationship  to  rest  of  system 

4.  Count  only  requirements  at  the  level  of  the  system  of  interest 

5.  Assess  complexity  of  requirements 


seari.mit.edu 
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SErVlnputs  Needed  to  Implement  COSYSMO 

Systems  Engineering  Advancement  Research  Initiative  1  1 


Effort  Multipliers 

Requirements  understanding 
Architecture  understanding 
Level  of  service  requirements 
Migration  complexity 
Technology  risk 

Documentation  to  match  life  cycle 
Tool  support 


#  and  Diversity  of  installations/platforms 

#  of  Recursive  levels  in  the  design 
Stakeholder  team  cohesion 
Personnel/team  capability 
Personnel  experience/continuity 
Process  capability 

Multisite  coordination  needs 


Data  source:  input  from  high-level  IPT. 
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Application  of  COSYSMO  to  HSI 


Systems  Engineering  Advancement  Research  Initiative 


HSI  requirements  include,  but  are  not  limited  to,  any 
requirement  pertaining  to  one  or  more  domains  of  HSI,  or 
the  integration  of  those  domains.  Broadly,  the  term 
encompasses  any  requirement  that  contributes  to  the 
integration  of  human  considerations  into  the  system  being 

developed. 


seari.mit.edu 


©  2009  Massachusetts  Institute  of  Technology 


11 


Systems  Engineering  Advancement  Research  Initiative 


Application  of  COSYSMO  to  HSI 

Notional  Example 


Shall’s  +  Will’s  +  Must’s 

450 
400 


} 

> 


Human  factors.  Human  factors  engineering  principles 
such  as  specified  in  MIL-STD-1472  shall  be  employed 
in  each  XXX  system  solution  (Threshold  =  Objective). 

^<-“hard” 

— *  “hard” 

<-“nominal” 

— >■  “nominal” 

<-“easy” 

>  easy 

> 

t 

Human  Interfaces 

seari.mit.edu 
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Systems  Engineering  Advancement  Research  Initiative 


Application  of  COSYSMO  to  HSI 

Notional  Example 
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Shall’s  +  Will’s  +  Must’s 


1.1 


5-Mar-07 


<S>  2007  Ricardo  Valerdi 


3  ENTER  SIZE  PARAMETERS  FOR  SYSTEM  OF  INTEREST 


Easy 

Nominal 

Difficult 

#  of  System  Requirements 

100 

200 

50 

#  of  System  Interfaces 

#  of  Algorithms 

#  of  Operational  Scenarios 

500 

0 

0 

0 

500 


equivalent  size 


10  SELECT  COST  PARAMETERS  FOR  SYSTEM  OF  INTEREST 
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20 


21 


22 


23 

24 
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Requirements  Understanding 

N 

1.00 

Arch  i  te  ctu  r e  U  nd  e  rsta  nd  i  ng 

N 

1.00 

Level  of  Service  Requirements 

H 

Migration  Complexity 

N 

1.00 

Technology  Risk 

N 

1.00 

Documentation 

N 

1.00 

#  and  diversity  of  installations/platforms 

N 

1.00 

#  of  recursive  levels  in  the  design 

N 

1.00 

Stakeholder  team  cohesion 

N 

1.00 

Personnel/team  capability 

H 

0.81 

Personnel  experience/continuity 

L 

Process  capability 

N 

1.00 

Multisite  coordination 

N 

1.00 

Tool  support 

N 

1.00 

composite  effort  multiplier 


26 


27 

28 


SYSTEMS  ENGINEERING  PERSON  MONTHS 


238  Person-Months 
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Application  of  COSYSMO  to  HSI 

Notional  Example 


Paste 
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CONSTRUCTIVE  SYSTEMS  ENGINEERING  COST  MODEL  _ 

<8>  2007  Ricardo  Valerdi 

ENTER  SIZE  PARAMETERS  FOR  SYSTEM  OF  INTEREST 


Easy 

Nominal 

Difficult 

#  of  System  Requirements 

50 

35 

25 

#  of  System  Interfaces 

#  of  Algorithms 

#  of  Operational  Scenarios 

185 

0 

0 

0 

185 


s  equivalent  size 


SELECT  COST  PARAMETERS  FOR  SYSTEM  OF  INTEREST 


Requirements  Understanding 

N 

1.00 

Architecture  Understanding 

N 

1.00 

Level  of  Service  Requirements 

H 

1.32 

Migration  Complexity 

N 

r  i.oo  i 

Technology  Risk 

H 

1.32 

Documentation 

N 

1.00 

#  and  diversity  of  installations/platforms 

N 

1.00 

#  of  recursive  levels  in  the  design 

N 

1.00 

Stakeholder  team  cohesion 

N 

1.00 

Personnel/team  capability 

N 

1.00 

Personnel  experience/continuity 

L 

Process  capability 

N 

1.00 

Multisite  coordination 

N 

1.00 

Tool  support 

L 

SYSTEMS  ENGINEERING  PERSON  MONTHS 


|  composite  effort  multiplier 


156  Person-Months 
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Systems  Engineering  Advancement  Research  Initiative 


Application  of  COSYSMO  to  HSI 
Application  to  HSI  Domains 


Habitability  Qc  Hea|th 

Safety  |  j  y-  Environment 
Personnel 


Manpower 


Survivability 


ESOH 


Training 


HSI-related  requirements  found  in  Government- 
furnished  requirements  documents 


How  complex  are  the 
safety  requirements? 

What  tools  are  available 
for  survivability 
analyses? 

How  verifiable  are  the 
environmental 
requirements? 

How  do  these  factors 
affect  level  of  effort? 
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Systems  Engineering  Advancement  Research  Initiative 


Application  of  COSYSMO  to  HSI 

Takeaways 


Procurement 


=  SE/PM  Costs 


seari.mit.edu 
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Systems  Engineering  Advancement  Research  Initiative 


Types  of  studies 

*  Technology  assessment 

*  Engine  size  and  cycle 

*  Design  life  optimization 

*  Stage  count,  configurate 
rotor  speeds 

*  Evolution  trom  demonstr; 
to  prototype 

Tradeoff  alternatives 

Design  and  programmatic 
alternatives  based  on 
cost,  schedule,  and 
performance  requirements 


HSI  Already  Integrated  Into  Systems 

Engineering 


Evaluation  criteria 

*  Safely 

*  Weapon  system  fife 
cycle  cost 

*  Sup  portability 

*  Reliability/maintainability 

*  Weight 

*  Operability, Stability 

*  Manpower,  personnel, 

and  training 


F119  Engine 

On  time 
Within  Cost 
Superior  Performance 


Planned  Iradesfudies 

*  AlfouJability 

*  Design  refinement 

*  Pre-planned  product 
improvement 

*  Materials  and 
manulacturing 
technology 


Balanced  design 


•  Low-risk 

•  Aflordable 

•  Achieves  all  ATF  / 
NATF  reauirements 


Yankel,J.,  &  Deskin,  W.  (2002).  "Development  of  the  F-22  propulsion  system." 
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Systems  Engineering  Advancement  Research  Initiative 


“Essentially,  all  models  are  wrong, 
but  some  are  useful” 

George  E.  P.  Box,  statistician 
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SE^V  How/When  to  Use  COSYSMO  for  HSI 

Systems  Engineering  Advancement  Research  Initiative 


Use  #1 :  “Are  my  ballpark  estimates  of  SE/PM  and  HSI  reasonable?” 

Required  Inputs: 

Existing  SE/PM  Cost  Estimate  (rule-of-thumb,  analogy,  etc.) 

Existing  draft  requirements  document 

IPT  or  expert  analysis  of  requirements  to  identify  HSI  Requirements 

Optional:  High-level  DoDAF  views  for  Operational  Scenarios 

Useful  Outputs: 

Identification  of  SE/PM  and  HSI  cost  drivers 

Identification  of  major  issues  (risk,  technology  maturity,  difficulty) 

Early  warnings  of  large  discrepancies 

Application: 

Pre-Milestone  A 

Can  use  any  available  information  (DoDAF,  Draft  Requirements,  etc.) 
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SE^V  How/When  to  Use  COSYSMO  for  HSI 

Systems  Engineering  Advancement  Research  Initiative 


Use  #2:  “What  is  the  right  amount  of  SE/PM  and  HSI  for  my  system?” 

Required  Inputs  (in  addition  to  Use  #1): 

Calibration  using  cost  data  from  previous  systems 
Decomposition  of  requirements  into  “sea-level”  requirements  or 
interfaces 

Useful  Outputs  (in  addition  to  Use  #1): 

Better  understanding  of  relative  effort  impact  of  each  requirement 
Improved  cost  estimate  compared  to  traditional  methods 

Application: 

Full  System  Life  Cycle 

Can  be  constantly  updated  in  response  to  new  information  or 
external  pressures 
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Future  Work 


Systems  Engineering  Advancement  Research  Initiative 


Recently  Developed  Resources  Useful  for  Implementation  of 
COSYSMO  for  HSI 


y.S.  AIR  FORCE 

Human  Systems  Integration  Office 


Human  Systems 
Integration  Requirements 

POCKET  GUIDE 


Human  Systems 
Integration  (HSI) 
in  Acquisition 


Integrating 
Human  Concerns 
into  Life  Cycle 
Systems  Engineering 


seari.mit.edu 
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An  Evidence-Based  Personnel  Competency 
Assessment  Framework  for 
Major  Defense  Acquisition  Programs  f M  DAPs) 


Barry  Boehm,  Dan  Ingold,  Winsor  Brown,  USC 

Paul  Componation,  UAH 
NDIA  Systems  Engineering  Conference 

29  October  2009 
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SYST E MS  ENG i N E BRING 

flesBarch  Center 


•  Competency  Assessment  Purposes  and  Models 

-  SERC  SE  Effectiveness  Measures  Scope  Decisions 

•  MDAP  SE  Competency  Assessment  Elements 

-  Evidence-based  SE  reviews  and  tools 

-  Early  life  cycle  concepts  of  operation 

-  SE  Competency  Assessment  Framework 

•  Results  of  Pilot  Evaluations 

•  Benefits  of  Usage 

•  Next  Steps  and  Research  Issues 

•  Conclusions 
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Competency  Assessment  Purposes  and  Models 


SYST E MS  ENG J N  E  ERl NG 

flesBarch  Center 


•  Personnel  Certification  Models 

—  Assess  degree  of  mastery  of  core  SE  knowledge,  skills,  abilities  (KSAs) 

—  Assessment  via  examination,  resume,  artifacts  produced 

•  Enterprise  KSA  Inventory,  Career  Progression  Models 

—  Record  degree  of  mastery  of  core  and  business-domain  SE  KSAs 
-  Assessment  via  educational  and  project  experience  records 

•  Project  SE  Staffing  Capability  Models 

—  Assess  commitment  to  provide  project-critical  skills 

•  Tailorable  subset  of  core  SE  skills 

•  Extendable  for  project-specific  skills 

—  Assessment  via  educational  and  project  experience  records,  interviews 
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flesBarch  Center 


Types  of  Milestone  Reviews 


•  Schedule-based  reviews  (contract-driven) 

—  We'll  hold  the  PDR  on  April  1  whether  we  have  a  design  or  not 

-  High  probability  of  proceeding  into  a  Death  March 

•  Event-based  reviews  (artifact-driven) 

—  The  design  will  be  done  by  June  1,  so  we'll  have  the  review  then 
—  Large  "Death  by  PowerPoint  and  UML"  event 

•  Hard  to  avoid  proceeding  with  many  unresolved  risks  and  interfaces 

•  Evidence-based  commitment  reviews  (risk-driven) 

—  Evidence  provided  in  Feasibility  Evidence  Description  (FED) 

•  A  first-class  deliverable 

•  Based  on  concurrently  engineered  ConOps,  specs,  and  plans 
—  Shortfalls  in  evidence  are  uncertainties  and  risks 

-  Should  be  covered  by  risk  mitigation  plans 

-  Stakeholders  decide  to  commit  based  on  risks  of  going  forward 
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SYST E MS  ENG J N E BRING 

flesBarch  C  e  n  t  e  r 


Content  of  Evidence-Based  Reviews 


•  Evidence  provided  by  developer  and  validated  by  ijidejjendent 
experts  that: 

If  the  system  is  built  to  the  specified  architecture,  it  will 
—  Satisfy  the  specified  operational  concept  and  requirements 
•  Capability,  interfaces,  level  of  service,  and  evolution 
—  Be  buildable  within  the  budgets  and  schedules  in  the  plan 
—  Generate  a  viable  return  on  investment 

—  Generate  satisfactory  outcomes  for  all  of  the  success-critical  stakeholders 

•  Shortfalls  in  evidence  are  uncertainties  and  risks 

-  Should  be  resolved  or  covered  by  risk  management  plans 

•  Assessed  in  increasing  detail  at  major  anchor  point  milestones 

-  Serves  as  basis  for  stakeholders'  commitment  to  proceed 

—  Serves  to  synchronize  and  stabilize  concurrently  engineered  elements 


Can  be  used  to  strengthen  current  schedule -  or  event-based  reviews 


SEPAT  Seeks  Performance  Evidence 

SYSTEMS  ENGINEERING 
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Reset 


NOTE:  Impact  and  evidence/risk  ratings  should  be  done  independently.  The 
impact  rating  should  estimate  the  effect  a  failure  to  address  the  specified  item 
might  have  on  the  program.  The  evidence  rating  should  specify  the  qualtity  of 
evidence  that  has  been  provided,  which  demonstrates  that  the  specified  risk  Risk 
item  has  been  satisfactorily  addressed.  Exposure 


Goal  1: 


Critical  Success  Factor  1.1 


Concurrent  definition  of  system  requirements  and  solutions 


1.1(e) 


Understanding  of  stakeholder  needs:  capabilities,  operational  concept,  key 
performance  parameters,  enterprise  fit  (legacy) 

r 

4 

At  Milestone  A,  have  the  KPPs  been  identified  in  clear,  comprehensive,  concise  terms  that 
are  understandable  to  all  stakehol  ders? 

No  forma 

Has  a  CONORS  been  developed  showing  that  the  system  can  be  operated  to  handle  both 
nominal  and  off-nominal  workloads,  to  meet  response  time  requirements,  and  generally 
to  meet  the  defined  KPPs? 

IT  system 

Has  the  ability  of  the  system  to  meet  mission  effectiveness  goals  been  verified  through 
the  use  of  modeling  and  simulation? 

IT  system 

effective  n 

Have  the  success-critical  stakeholders  been  identified,  their  roles  and  responsibilities 
negotiated,  and  their  needs  clearly  represented  by  the  KPPs  and  CONORS? 

□evelopnr 
Stake  hole 

Have  issues  about  the  fit  of  the  system  into  the  stakeholders'  context--  acquirers,  end 
users,  administrators,  interoperators,  maintained,  etc.  —  been  adequately  explored? 

Explored 
after  syst 
related  to 

different  1 
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SECAT  Seeks  Competency  Evidence 

That  can  bejndependently  validated 


SYST B MS  B NCj J N  B BRING 

Research  Center 
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NOTE:  Impact  and  evidence/risk  ratings  should  be  done  independently.  The 
impact  rating  should  estimate  the  effect  a  failure  to  competently  address  the 
specified  item  might  have  on  the  program.  The  competency  rating  should 
specify  the  observed,  historical  experience  and  competency  of  the  systems 
engineering  staff  on  past  programs  with  respect  to  the  specified  risk  item. 


Goal  1: 


Concurrent  definition  of  system  requirements  and  solutions 


4 

4 

3 

3 

4 


Critical  Success  Factor  1.1 


Understanding  of  stakeholder  needs:  capabilities,  operational  concept,  key 
performance  parameters,  enterprise  fit  (legacy).  Evidence  of  ability  to  analyze 
strengths  and  shortfalls  in  current-system  operations  via: 

Participatory  workshops,  surveys,  focus  groups? 

Operations  research  techniques:  operations  data  collection  and  analysis? 

Mission  effectiveness  modeling  and  simul  ation? 

Prototypes,  scenarios,  stories,  personas? 

Ethnographic  techniques:  Interviews,  sampled  observations,  cognitive  task  analysis? 


Reset 


Risk 

Exposure 
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SYST E MS  ENG i N E BRING 

flesBarch  Center 


Outline 


•  Competency  Assessment  Purposes  and  Models 
—  SERC  SE  Effectiveness  Measures  Scope  Decisions 

•  MDAP  SE  Competency  Assessment  Elements 
—  Evidence-based  SE  reviews  and  tools 

—  Early  life  cycle  concepts  of  operation 
— ^  —  SE  Competency  Assessment  Framework 

•  Results  of  Pilot  Evaluations 

•  Benefits  of  Usage 

•  Next  Steps  and  Research  Issues 

•  Conclusions 
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svs tk™*k™SE  Effectiveness  Measurement  Methods  Used 

Research  Center 


•  NRC  Pre-Milestone  A  &  Early-Phase  SysE  top-20  checklist 

•  Services  Probability  of  Program  Success  (PoPS)  Frameworks 

•  INCOSE/LMCO/MIT  Leading  Indicators 

•  Stevens  Leading  Indicators  (new;  using  SADB  root  causes) 

•  USC  Anchor  Point  Feasibility  Evidence  progress 

•  UAH  teaming  theories 

•  NDIA/SEI  capability/challenge  criteria 

•  SISAIG  Early  Warning  Indicators/  USC  Macro  Risk  Tool 
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SYST E MS  ENG J N E BRING 

flesBarch  C  e  n  t  e  r 


Additional  Personnel  Competency  Sources 


ASN  (RD&A),  Guidebook  for  the  Acquisition  of  Naval  Software- 
Intensive  Systems,  Version  1.0,  September  2008 

L.  Bass  et  al.,  Models  for  Evaluating  and  Improving  Architecture 
Competence,  CMU/SEI-2008-TR-006,  April  2008 

INCOSE  Systems  Engineering  Handbook,  INCOSE-TP-2003-002- 
03.1 ,  August  2007 

ODNI,  Subdirectory  Data  Collection  Tool:  Systems  Engineering, 
2008. 

R.  Pew  and  A.  Mavor,  Human-System  Integration  in  the  System 
Development  Process:  A  New  Look,  National  Academies  Press, 
2007. 

C.  Williams  and  M.  Derro,  NASA  Systems  Engineering  Behavior 
Study,  NASA  Office  of  the  Chief  Engineer  October  2008. 
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Initial  EM  Coverage  Matrix 


SERC  EM  Task  Coverage  Matrix  VI. 0 


NRC 

Probability  of 

SE  Leading 

LIPSF 

Anchoring  SW 
Process 
(USC) 

PSSES 

SSEE 

Macro  Risk 

Success 

Indicators 

(Stevens) 

(U.  of  Alabama) 

(CMU/SEI) 

Model/Tool 

Concept  Dev 

Atleast  2  alternatives  have  been  evaluated 

X 

X 

X 

X 

(w.r.tNPR) 

(x) 

Can  an  initial  capability  be  achieved  within  the  time 
that  the  key  program  leaders  are  expected  to 
remain  engaged  in  their  current  jobs  (normally  less 

X 

(X) 

than  5  years  or  so  after  Milestone  B)?  If  this  is  not 

x 

(X) 

(5  years  is  not 

(seems  to  be 

(x) 

possible  for  a  complex  major  development 

explicitly 

inferrable  from 

(implies  this) 

program,  can  critical  subsystems,  or  at  least  a  key 
subset  of  them,  be  demonstrated  within  that  time 
frame? 

stated) 

the  conclusions) 

Will  risky  new  technology  mature  before  B?  Is  there 

(x) 

a  risk  mitigation  plan? 

Have  external  interface  complexities  been 
identified  and  minimized?  Is  there  a  plan  to 
mitigate  their  risks? 

X 

X 

X 

X 

X 

X 

KPP  and  CONOPS 

At  Milestone  A,  have  the  KPPs  been  identified  in 
clear,  comprehensive,  concise  terms  that  are 
understandable  to  the  users  of  the  system? 

X 

(X) 

X 

(X) 

X 

(strongly 

implied) 

(x) 

(implied) 

X 

X 

At  Milestone  B,  are  the  major  system-level 
requirements  (including  all  KPPs)  defined 
sufficiently  to  provide  a  stable  basis  for  the 
development  through  IOC? 

X 

X 

(X) 

X 

X 

(x) 

(x) 

(There  is  no  direct 
reference  to  this 
but  is  inferrable) 

X 

(x) 

(there  is  a  mention 

Has  a  CONOPS  been  developed  showing  that  the 

X 

X 

(X) 

(X) 

X 

of  a  physical 
solution.  That's  the 

X 

X 

system  can  be  operated  to  handle  the  expected  closest  in  this 

throughput  and  meet  response  time  requirements?  regard) 

Legend: 

x  =  covered  by  EM 

(x)  =  partially  covered  (unless  stated  otherwise) 


Personnel  Competency: 
Commonality  of  Goal  Frameworks 
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SERC  EM  Framework 

NDIA  Personnel 
Competency  FW 

SEI  Architect 
Competency  FW 

Concurrent  Definition  of 
System  Requirements  & 
Solutions 

Systems  Thinking 

Stakeholder  Interaction 

System  Life  Cycle 
Organization,  Planning, 
Staffing 

Life  Cycle  View 

Other  phases 

Technology  Maturing  and 
Architecting 

SE  Technical 

Architecting 

Evidence-Based  Progress 
Monitoring  &  Commitment 
Reviews 

SE  Technical 
Management 

Management 

Professional/  Interpersonal 
(added) 

Professional/ 

Interpersonal 

Leadership,  Communication, 
Interpersonal 

Example  Personnel  Competency  Questions 


*m'  V*  * 

SYSTEMS  ENGINEERING 
flesBarch  Center 


1.  Concurrent  Definition  of  System  Requirements  &  Solutions 

1.1  Understanding  of  stakeholder  needs:  Capabilities,  Operational  Concept, 
Key  Performance  Parameters,  Enterprise  fit  (legacy).  Evidence  of  ability  to 
analyze  strengths  and  shortfalls  in  current-system  operations  via: 

a.  Participatory  workshops,  surveys,  focus  groups? 

b.  Operations  research  techniques:  operations  data  collection  and  analysis, 
modeling? 

c.  Prototypes,  scenarios,  stories,  personas? 

d.  Ethnographic  techniques:  Interviews,  sampled  observations,  cognitive  task 
analysis? 

1.2  Concurrent  exploration  of  solution  opportunities;  Analysis  of  Alternatives 
for  cost-effectiveness  &  risk  (Measures  of  Effectiveness).  Evidence  of 
ability  to  identify  and  assess  alternative  solution  opportunities  via 
experimentation  and  analysis  of: 

a.  Alternative  work  procedures,  non-materiel  solutions? 

b.  Purchased  or  furnished  products  and  services? 

c.  Emerging  technology? 

d.  Competitive  prototyping? 


SYST E MS  ENG i N E BRING 

flesBarch  Center 


Outline 


•  Competency  Assessment  Purposes  and  Models 

-  SERC  SE  Effectiveness  Measures  Scope  Decisions 

•  MDAP  SE  Competency  Assessment  Elements 

-  Evidence-based  SE  reviews  and  tools 

-  Early  life  cycle  concepts  of  operation 

-  SE  Competency  Assessment  Framework 
— ^  •  Results  of  Pilot  Evaluations 

•  Benefits  of  Usage 

•  Next  Steps  and  Research  Issues 

•  Conclusions 
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Pilot  Feedback  Highlights 


SY5T E MS  ENG J N E BRING 

flesBarch  C  e  n  t  e  r 


•  Primarily  useful  during  early  stages 

—  SEPAT:  Tech  Development,  60%;  System  Development,  100% 

—  SECAT:  Tech  Development,  50%;  System  Development,  75% 

—  Between  "Very  Effective"  and  "Somewhat  Effective" 

•  Too  many  Red  and  Yellow  risks 

—  Rating  scales  reworked 

•  Overly  DoD-specific  (NASA  responder) 

•  Need  versions  for  different  domains,  project  types 

—  Quick-response/agile;  legacy-driven;  KPP-driven;  sea;  space; ... 

•  Make  question  format  uniform  across  SEPAT  and  SECAT 
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SYST B MS  B NCj J N  B BRING 

Research  Center 


EM  Processes  end  Tools  Help  Enable 
MDAP  Transformation 


Implements  spirit  of  July  2009  Augustine  BENS  Report 


Adversarial  Mistrust  Collaborative  Trust-and-Verify 


SYST E MS  ENG J N  E  BRING 

flesBarch  Center 


Project  and  Tool  Status  and  Plans 


•  We  have  two  tools  for  evaluating  systems  engineering  (SE) 
effectiveness  in  the  definition  and  development  stages  of  Major 
Defense  Acquisition  Programs 

—  SE  Performance  Assessment  Tool  (SEPAT) 

—  SE  Capability  Assessment  Tool  (SECAT) 

—  Based  on  analysis  and  synthesis  of  major  sources  of  DoD  SE  EMs 

—  Including  concepts  of  operation  for  project  usage,  sponsor- 
developer  coordination,  SE  EM  knowledge  base  development 

•  We  have  piloted  the  tools  on  (7  now;  over  12  expected)  projects 

—  And  evaluated  them  with  respect  to  the  ODDR&E-SSE  Systemic 
Analysis  Database  (SADB) 

—  Feedback  mostly  positive;  some  good  improvement  suggestions 

•  We  are  incorporating  some  suggestions  and  have  drafted  plans 
for  followon  improvement  efforts 
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Research  Center 


Bottom  Line  Message 


SE  shortfalls  are  a  major  source  of  DoD  system  acquisition  problems 

—  Systemic  Analysis  Database  analysis  results 

SE  EM  shortfalls  are  a  major  source  of  SE  effectiveness  problems 

—  You  can't  control  what  you  can't  measure 

The  SECAT  and  SEPAT  tools  enable  a  measurement-driven  SE  process 

—  Via  negotiated  MDA-acquirer-developer  EM-based  approach 

EM-driven  SE  improvement  has  high  ROI  for  MDAPs 

—  ROI  varies  with  system  size,  criticality,  volatility 

The  SERC  SE  EM  tools  are  approaching  general-use  maturity 

—  Core  tools  are  in  the  TRL  5-6  (alpha-beta  test)  range 

—  Domain/life  cycle  extensions,  risk  summaries,  mitigation  guidance  TBD 

Draft  plan  to  mature,  extend,  transition  technology  in  work 

Looking  for  collaborators,  early  adopters  interested  in  reducing  their 
09/o9Y§jTun  and  delivery  shortfall  rates 
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The  SERC  EM  Team 
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•  USC:  Barry  Boehm,  Dan  Ingold,  Winsor  Brown,  JoAnn  Lane, 
George  Friedman 

•  Fraunhofer-Maryland:  Kathleen  Dangle,  Linda  Esker,  Forrest 
Shull 

•  Stevens:  Rich  Turner,  Jon  Wade,  Mark  Weitekamp 

•  U.  Alabama-Huntsville:  Paul  Componation,  Sue  O'Brien,  Dawn 
Sabados ,  Julie  Fortune 
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Summing  of  Major  Scope  Decisions 
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Decision 

*  MDAP  vs.  multi-type  EMs  • 

*  Core  vs.  all-domain  EMs  • 

*  Ease  of  tailoring,  extension  • 

*  Cover  SE  functional  performance  • 

and  personnel  competency 

*  Rate  both  degree  of  impact  and 
degree  of  satisfaction  evidence 

*  Hierarchical  goal  -  critical  success  * 

factor  -  question  framework 

*  Compatibility  with  INCOSE  * 

Leading  Indicators 

*  Framework  and  tools  • 

*  Pilot  use  and  evaluation  • 

*  Initial  focus  on  project  assessment  * 

vs.  practice  ROIs 


Rationale 

SE  shortfalls  a  major  MDAP  problem 
Avoid  numerous  inapplicable  EMs 
Enable  special-community  tailoring 
Sponsor  priority 

Relation  to  risk  exposure  RE=P(L)*S(L),  ease 
of  tailoring  out  zero-impact  questions 
Ease  of  use,  understanding;  compatibility 
with  related  frameworks 
Complementary  coverage:  continuous  vs. 
discrete;  quantitative  vs.  qualitative 
Early  SERC  tangible  product 
Evidence  of  strengths  and  shortfalls 
ROI  data  unavailable;  could  be  generated 
via  tool  use 


SYSTEMS  ENGINEERING 
Research  Center 


Magnitude  of  MDAP  Problem 


Analysis  of  U.S.  Defense  Dept. 

Major  Defense  Acquisition  Program  Portfolios 

Fiscal  2009  dollars 


Portfolio  size 

Number  of  programs 
Total  planned  commitments 
Commitments  outstanding 


2003  2007  2008 


77 

$1.2  trillion 
$724.2  billion 


95 

$1.6  trillion 
$875.2  billion 


96 

$1.6  trillion 
$786.3  billion 


Portfolio  indicators 

Change  to  total  RDT&E*  costs  from  first 
Change  to  total  acquisition  cost  from  first 
Total  acquisition  cost  growth 
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Average  schedule  delay  in  delivering  initial  capabilities  18  months  21  months  22  months 


Source:  U.S,  Government  Accountability  Office  *  Research,  Development ,  Testing  &  Evaluation 
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Operational  concepts  for  EM  tool  usage 


SYST E MS  ENG i N E BRING 

flesBarch  Center 


•  EM  tools  used  to  reach  sponsor-performer  consensus  on 
way  forward 

—  Via  EM-based  risk  assessments 

•  Three  scenarios 

—  Milestone  A:  Acquirer  and  Milestone  Decision  Authority  (MDA) 

•  MDAP  and  non-MDAP  cases 

—  Contract  Negotiation:  MDAP  Acquirer  and  Developer 
—  Project  Execution:  MDAP  Developer  Manager  and  Performers 
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SYST E MS  ENG INEEBi NG  Scenario  1.  Acquirer  and  MDA  at  Milestone  A 

Research  Center  1 

•  Acquirer  submits  proposed  acquisition  plan  to  MDA  with  SEPAT, 
SECAT  ratings  and  risk  mitigation  approaches 

•  MDA  has  independent  experts  review  SEPAT,  SECAT  ratings 

—  Major  finding:  Analysis  of  Alternatives  rated  No  Impact,  no  risk 
—  MDA  asks  Acquirer  for  AoA  impact  rationale 

•  Acquirer  response:  Case  1 

—  Capability  is  needed  quickly  for  limited  but  critical  use 
—  Evidence  is  available  that  Alternative  A  solution  is  sufficient 
—  MDA  response:  Rationale  is  sufficient.  OK  to  proceed 

•  Acquirer  response:  Case  2 

—  DARPA  demo  has  shown  proof  of  principle.  All  that  is  needed  is  to 
implement  it  for  the  general  case 

—  MDA  response:  No  evidence  of  scalability,  ability  to  handle  degraded 
battle  conditions.  Resubmit  using  Competitive  Prototyping 

09/08/2009  25 
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Competitive  Prototyping  Btntfits  Example  -  4:1  RPV 


•  Total  Commitment 

-  Agent  technology  demo  and  PR:  Can  do  4:1  for  $1B 

—  Winning  bidder:  $800M;  PDR  in  120  days;  4:1  capability  in  40  months 
—  PDR:  many  outstanding  risks,  undefined  interfaces 

-  $800M,  40  months:  "halfway"  through  integration  and  test 

-  1:1  IOC  after  $3B,  80  months 

•  CP-based  Incremental  Commitment  [number  of  competing  teams] 

-  $25M,  6  mo.  to  VCR  [4]:  may  beat  1:2  with  agent  technology,  but  not  4:1 

-  $75M,  8  mo.  to  ACR  [3]:  agent  technology  may  do  1:1;  some  risks 

-  $225M,  10  mo.  to  DCR  [2]:  validated  architecture,  high-risk  elements 

-  $675M,  18  mo.  to  IOC  [1]:  viable  1:1  capability 

-  1:1  IOC  after  $1B,  42  months 


Scenario  2.  Acquirer-Developer 
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•  Acquirer  tailors  SEPAT,  SECAT  to  project  specifics 

—  Domain  and  project  extensions 
—  Question  impact/priority  ratings 

•  Acquirer  coordinates  SEPAT,  SECAT  usage  with  developer 

—  As  mutual  instruments  for  monitoring  SE  effectiveness 
—  At  major  milestones  and  project  reviews 
—  Portion  of  award  fee  based  on  review  of  evidence 

•  Developer  analyzes  implications  for  project  SE  effort 
—  Options  on  evidence  production,  associated  costs 

•  Developer,  Acquirer  converge  on  options 

—  And  adjustments  to  questions,  impact  ratings,  SE  budgets, 
milestone  content,  contract  provisions 
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Scenario  2  Example 

•  Acquirer  specifies  CSF  1.2(d)  to  have  Critical  impact: 

—  Have  the  claimed  quality  of  service  guarantees  been  validated? 

•  Winning  competitive  prototyping  developer  responds: 

—  This  would  be  incompatible  with  your  proposed  contract,  which 
ties  our  System  Functional  Requirements  Review  milestone 
progress  payments  and  award  fees  to  specifying  functionality. 
Our  proposed  SE  plans  and  budgets  don't  cover  doing  QoS 
guarantees  by  then. 

•  Acquirer  responds: 

—  Thanks.  The  contract  clearly  undercuts  our  intent  to  do 

evidence-based  concurrent  engineering,  and  sets  us  up  for  late 
overruns.  We'll  redo  it  and  your  SE  plans  and  budgets.  Next 
time,  we'll  address  contracting  compatibility  earlier. 
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Scenario  3.  Project  EM 


•  *♦’  ti« 

SYSTEMS  ENGINEERING 
Research  Center 


•  Primary  responsibilities,  authority,  accountability  (RAA) 

—  Primary  assessment  consumers:  Persons  with  management 
responsibility  for  program  results 

•  Contractor  PM,  DoD  acquirer  PM/PEO,  oversight  personnel 

—  Primary  assessment  conveners,  monitors:  Chief  Engineers, 
Chief  Systems  Engineers 


—  Primary  assessors:  Independent  experts 
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Project  SysE  EM  Operational  Concept 

tfer  taeh  stigt  e>f  system  dtfiimtiiQif)  indi  dtvtl<apmtnt), 


Develop/update 

Evaluate  staffing  plans 

>  SEMP, 

- ► 

vs.  SECAT. 

SEP,  including 
SE  staffing  plans 

Evaluate  rest  of  SEMP, 

SEP  vs.  SEPAT 

Execute  opportunistic 
development 


Yes 


0" 

Set  INCOSE 
Leading  Indicators 
Control  Limits 

— ► 

Execute 

Program 

Detailed  EM  assessment(s), 
corrective  action 


Evaluate  staffing  plans 
vs.  SECAT 

Evaluate  rest  of  SEMP, 
SEP  vs.  SEPAT 


Corrective  action 


J 
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First-Order  EM  Evaluation  Process 
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•  Examine  revised  list  of  candidate  EMs 

—  Use  NRC  early  SE  checklist  as  concise  starting  point 
—  Identify  similar  key  elements  of  other  EMs 
—  45x8  cross  product  of  EMs  and  characteristics 

•  Evaluate  EMs  against  identified  criteria 

—  Preliminary  "quick-look"  evaluation  by  USC 
—  Evaluation  by  originators,  where  possible 
—  Follow-up  with  independent  evaluation  by  team 

•  Review  coverage/commonality  of  elements 

—  Incorporate  suggested  additions  (now  51  items) 


Structuring  the  51  EM  Elements 


SYST E MS  ENG J N E BRING 
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Systems  Engineering  Effectiveness  Measurement 
Proposed  New  Framework 

SEPP-Guide- 

BasedEval. 

Framework 

SISAIG / 
Macro  Risk 
Framework 

Coverage 
Matrix  Items 

1.  Concurrent  Definition  of  System  Requirements  &  Solutions 

1.1  Understanding  of  stakeholder  needs:  Capabilities, 
Operational  Concept,  Key  Performance  Parameters, 
Enterprise  fit  (legacy) 

1.1,  1.4, 
3.1 

1.1,  1.4 

5, 7, 22, 
36,37 

1.2  Concurrent  exploration  of  solution  opportunities;  AoA’s  for 
cost-effectiveness  &  risk  (Measures  of  Effectiveness) 

4.1,4.2 

1.2 

1,14,26, 

27,28 

1.3  System  scoping  &  requirements  definition  (External 
interfaces;  Memoranda  of  Agreement) 

1.2, 1.4 

3.2 

4  6,  13, 

50 

1.4  Prioritization  of  requirements  &  allocation  to  increments 

1.3 

1.5 

2,11,31 
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Steps  for  Developing  Feasibility  Evidence 


A.  Develop  plans  for  developing  work-products/artifacts 

B.  Determine  most  critical  feasibility  assurance  issues 

—  Based  on  SEPAT,  SECAT  question  impact/priority  ratings 

C.  Evaluate  feasibility  assessment  options 

—  Cost-effectiveness,  rework  avoidance ,  risk  reduction  ROI 
—  Tool,  data,  mission  scenario  availability 

D.  Select  options,  develop  feasibility  assessment  plans 

E.  Prepare  evidence  development  plans  and  earned  value 
milestones 


09/08/2009 
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to  indicate  that  many  are  done  concurrently 
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systrms eng.nf.ering  Steps  for  Developing  Feasibility  Evidence  (cont.) 

Research  Center 


F.  Begin  monitoring  progress  with  respect  to  plans 

—  Also  monitor  project/technology/objectives  changes  and  adapt  plans 

G.  Prepare  evidence-generation  enablers 

—  Assessment  criteria 

-  Parametric  models,  parameter  values,  bases  of  estimate 

-  COTS  assessment  criteria  and  plans 

-  Benchmarking  candidates,  test  cases 

—  Prototypes/simulations,  evaluation  plans,  subjects,  and  scenarios 
—  Instrumentation,  data  analysis  capabilities 

H.  Perform  pilot  assessments;  evaluate  and  iterate  plans  and  enablers 

I.  Assess  readiness  for  SEPAT-SECAT  evidence  assessment 

—  Evidence  shortfalls  identified  as  risks  and  covered  by  risk  mitigation  plans 
—  Proceed  to  Milestone  Review  if  ready 

J.  Hold  Milestone  Review  when  ready;  adjust  plans  based  on  review 
outcomes 
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SE  Performance,  Competency  are  Major  Sources 
qsq/^j^l  Systemic  Analysis  Negative  Findings 


1  Technical  process  (35  instances) 
-  V&V,  integration,  modeling&sim. 

2  Management  process  (31) 

3  Acquisition  practices  (26) 

4  Requirements  process  (25) 

5  Competing  priorities  (23) 


6  Lack  of  appropriate  staff  (23) 

7  Ineffective  organization  (22) 

8  Ineffective  communication  (21) 

9  Program  realism  (21) 

10  Contract  structure  (20) 


Survivable  Vehicles  for  the  Warfighters 


Systems  Engineering  and 
Logistics:  Gunners  Restraint 


Systems  Engineering 
Conference 
29  October  2009 


Michelle  Bowen 

Chief  System  Engineer  for 
Logistics 


Integrated  Defense  Acquisition,  Technology,  and  Logistics  Life  Cycle  Management  System 
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Agenda 


❖  MRAP  Overview 

❖Systems  Engineering  and  Logistics 

❖Implementation  of  Systems  Engineering  to  save  lives: 
Gunners  Restraint  System  (GRS) 
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Tactical  Response 
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❖  Change  in  enemy 
tactics  generated 

an  urgent  Warfighter 
need  for: 

Mine  Resistant  Ambush 
Protected  Vehicle 

■  Large  quantities 

❖  MRAP  Program  is  the 
response  to  this 
urgent  need 

Unprecedented  effort 

■  Unprecedented  speed 
Unprecedented  Gov  / 

Industry  Teamwork 


Delivering  Survivable,  Jb: 
Fully  Capable 
Vehicles ...  r 
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. . .  With  Speed  and  Urg$jpcy\ 


#  MRAP 
Vehicles 


15,000 


10,000 


5,000 


Operational  Demand  Signal 


21,482,  Jun  09 
AT&L  ADM 


Service  & 
Congressional  call 
for  added  vehicle 
protection  drove 
rapid  requirements 
growth. 


15,374,  Sep  07 
JROC  validated 


16,238,  Nov  08 
JROC  validated 


Supported  M-ATV 
reqts  for  OEF 


15,838,  Jul  08 
JROC  validated 


7,774,  May  07 
JROC  validated 


6,738,  Feb  07 
MROC  validated 


4,066,  Nov  06 
Army-USMC  board 


1,185,  Dec  06 
JROC  Validated 


Increased  Army 
totals  from  2,500  to 
10,000  vehicles 
and 

included  100  test 
vehicles 


Supported 
increased  vehicle 
reqts  for  OEF 

Established  Army 
final  reqt  at  12K, 
SOCOM  final  reqt 
at  378  and 
final  ballistic  test 
reqt  at  1 33 
vehicles 


185,  May  06  MNF-W  Commander 
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MRAP  Vehicle  Variants 
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❖  To  meet  the  requirement  as  quickly  as  possible  5  IDIQ 
contracts  were  awarded. 


❖Requirements  were  written  to  the  capability  industry  had 
at  this  time 


GDLS  RG-31  CAT  l/EM 


BAE  SOCOM  (RG-33)  CAT  I 
/Plus 


Navistar  MaxxPro  CAT  I 
/Plus/Dash 


FPI  Cougar  CAT  I/Plus 


BAE-TVS  Caiman  CAT  1/ 
Plus 


BAE  RG-33L  CAT  I  I/Plus 


BAE  HAGA  CAT  I  I/Plus 


FPI  Cougar  CAT  I  I/Plus 
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Systems  Engineering  and  Logistics 


❖  As  we  support  the  operations  in  OIF/OEF  it  is  critical  that  we  continue 
to  integrate  logistics  into  our  SE  process  closely 

^Specific  processes  are  in  place  bringing  logistics  in  at  the  front  end  of 
engineering  decisions 

■  Requirements  Decomposition 

■  Design 

■  Feedback  from  the  Warfighter 


/ - 

Systems  Engineering  and  Logistics  Cannot  Be  Separated 
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MRAP  Integrated  SE  and  Logistics 


■ 


❖Only  PdM  with  a  Chief  Systems  Engineer  for  Logistics 

❖PdM  Logistics  has  implemented  a  planning  cell  that  uses 
SE  processes  to  accomplish  each  levied  requirement 

"  Integrated  Master  Schedule/Critical  Path  analysis 

■  Risk  analysis  and  Mitigation  Strategies 

■  Roles  and  Responsibilities 

■  End  to  end  Life  Cycle  Analysis  (example:  follow-on  roll  over  trainers) 


❖Logistics  is  a  heavily  weighted  factor  in  the  requirements 
management  process  to  include  commonality,  install  level 
and  theater  of  operation 
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GRS  SE/LOG  that  Saves  Lives 


Requirement/Requirements  Analysis: 

■  MRAP  vehicles  are  more  prone  to  rollover  due  to  their  weight  and 
higher  center  of  gravity 

>  Gunners  were  being  thrown  from  the  vehicle  due  to  rollover 

■  Sept.  20,  2008  during  a  visit  to  Bagram  Air  Field  in  Afghanistan 
Secretary  of  the  Army  Pete  Geren  was  informed  that  the  MRAP 
didn’t  have  a  gunners  restraint  like  other  tactical  vehicles. 

>  All  MRAPs  received  a  GRS 

>  All  GRSs  were  fielded  by  February  1  2009 
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Design/Integration/Verification/Validation 


❖  24  September  2008  RDECOM  was  tasked  to  design  a 
GRS  for  MRAP  in  72  hours. 

■  Integration  of  the  existing  gunners  restraint  of  the  BAE  Ml  114 

■  Systematic  approach  to  designing  the  minimum  amount  of  A-Kits  for  9 
different  variants. 

■  No  CAD  of  any  of  the  MRAP  vehicles. 

^Design  Completed  and  User  Jury  conducted  at 
TARDEC  before  sending  to  test  at  ATC 


❖  ILSC  tech  writer  sitting  side  by  side  with  designers  to 
produce  install  instructions  and  parts  and  special  tools 
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Design/Integration/Verification/Validation,  cont. 


❖  Upon  arrival  at  ATC,  redesign  was  required 
due  to  a  human  factors  issue 

❖  All  parties  pulled  together  to  analyze  the 
issue,  redesign/update  the  drawings  and  prototype  the 
part  for  test  over  night. 


❖  Testing  completed  on  3  variants  by  Sept  27  allowing 
production  to  begin  at  Rock  Island  Arsenal  and  Blue 
Grass  Army  depot.  The  remaining  variants  following 
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1  MRAP  j 

GRS  Timeline 
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Summary 


❖  MRAP’s  use  of  systems  engineering  principles  in 
logistics  lead  to  fielding  MRAP  as  quickly  as  possible. 

❖  Integrating  Logistics  into  the  SE  process  early  is  critical 
to  support  the  Warfighter 
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MRAP  “the  Ultimate  Team  Sport” 


BAE  SYSTEMS 


Questions? 
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Engineering 
(  EPIJ-  Process 
\  /^  Improvement 

Lean  Advancement  Initiative 

Enhancing  Systems 
Engineering  Competencies  in 

the  Enterprise 

29  October  2009 


LOCKHEED 


Garry  Roedler 


©  2009  Lockheed  Martin  Corp. 


10/19/2009 


Objective  of  Presentation 


•  Communicate  the  elements  of  the 
Engineering  Professional 
Development  program  for  Systems 
Engineering  at  Lockheed  Martin. 


©  2009  Lockheed  Martin  Corp. 


Vision 


A  comprehensive  set  of  skills  and  a 
curriculum  that  is  integrated  across 
disciplines  to  provide  the  foundation  for 
engineering  professional  development  and 
qualification,  and  enable  flexible  career 
paths. 


A  br 
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Corporate  Technical  Learning  Council 

Role 

■  Integrates  the  efforts  of  all  Business  Areas  and  Corporate 
Organizations  involved  with  technical  learning 

Function 

■  Communication  and  coordination  forum 

■  Promotes  teamwork  and  cooperation 

Goals 

-  Reach  more  of  the  workforce 

■  Improve  learning  effectiveness 

■  More  effective  and  motivated  workforce 

■  Higher  retention  levels 

■  Greater  recruiting  discriminators 
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Integration  Drivers 

*  CTLC  identified  needs  and  set  vision  for  Engineering  Professional 
Development 

-  Need  to  address  eroding  technical  base  and  preserve  knowledge 


•  EPD  VSM  focused  on  overall  strategy  for  Engineering  Professional 
Development 


From  “Cylinders  of 
Excellence”  with 
Separate  Assets  to  ... 


Integrated  Approach 
Using  a  Common 
Set  of  Assets 


Objectives 

•  Same  “look  and  feel” 

•  Allow  identification  of  common 
Skills  and  Training  needs 

•  Promote  consistent 
understanding  of  concepts, 
terms,  etc. 

•  Facilitate  cost-effective  course 
development  via  common 
courses,  where  applicable 

•  Framework  for  common 
engineering  needs  along  with 
discipline  specific  needs 


integration 


©  2009  Lockheed  Martin  Corp. 


10/29/2009  5 


Integrated  Approach  to  Address 
Skills,  Training,  and  Career  Path 


Recommended  delivery 
method(s)  for  courses 


Single  Development/ 
Qualification  Guide 
>Single  approach 
>Common  terminology 
>  Appendices  for 
supplemental  information 
for  each  discipline/  role 
>Provides  for  single 
communication  effort 


3d  Skill < 
!  skill  arc 
ion  term 
in  to  she 
ation  to  < 
line/  role 
s  identifi 
i  skill  re< 


BA/BU  Needs  &  Requirements 
Product  &  Implementation  Plans 


Integrated  Curriculum 
identifies  and  defines 
common  courses 
includes  discipline  unique 
and  specialization  courses 
identifies  applicability  of 
courses  to  disciplines/roles 
> Facilitates  greater  leverage 
among  disciplines 
>Curriculum  includes  the 
following  information  about 
each  course: 

Description/abstract 
Annotated  outlines 
Learning  objectives 
Audience 
Pre-requisites 
Level  of  Course 

path 


Ei 

requirements 


ung 

e 

am 


Dn 

m 

>le 


Engineering  Development  and 
Qualification  Program  (EDQP) 


Framework  to  develop,  verify  and  recognize  the 
knowledge,  experience  and  capabilities  of  practicing 
engineers 

-  Establishes  common  expectation  of  the  specific 
engineering  capabilities 

-  Facilitates  technical  development  and  career  path 
planning  of  engineers  (including  those  new  to  the 
discipline) 


-  Defines  capabilities  and  experiences  for  use  by  HR  & 
leaders  to  develop  staffing  plans/execute  staffing 

Builds  on  documented  skills  and  curriculum 


Includes  multiple  stages  of  development 


©  2009  Lockheed  Martin  Corp. 


10/29/2009  7 


BTtlL 


Identify  L&D 
Direction 


Create  a  Tailorable 
Framework 


Encourage  Individual 
Responsibility 
for  Development 


Provide  Enabling 
Resources 
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Key  EDQP  Elements 


Experience/OJT 

•  Discipline  &  domain 

•  Successful 
demonstration  of  skills 

Training/Education 

•  Consistent  foundation 
knowledge  per 
curriculum 

Coaching 

•  First  receiving 
coaching 

•  Later  providing 
coaching 

Mentoring 

•  First  as  Mentee 

•  Later  as  Mentor 

Basis  of  Qual  Criteria 


Skills  Portfolio 

(Competencies)  Qualification  Stage 

Criteria  per  Role 

Assessment 


Con-Ops  & 
Review  Board 

BA/BUs  Implement 
Tailored  Program 

Acknowledgement 
of  Qualification 

Sustainment  Rating 


Sysiynj-j'ijc  ? bfsosmul  Db'jzlop 
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EDQP  Development  Con-ops 


3-Assess 

Capabilities 


Coach 


7  -  Explore 
Options 


6  -  Perform  L&D 
Activities 


2  -  Evaluate 
Self- 

Assessment 


Employee 


Mentoj 


4  -  Validate 
Assessment  & 
Feedback 
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Talent 

Managers 


3-  Assess 
Capabilities 


Employee 


Leader 


4  -  Validate 

Assessment  &  Feedback 


EDQP  Qualification  Con-ops 


xxDQP  Review  Board 

Hi  HU 


7  -  Assess 
Stage  &  Feedback 


ir 


8  -  Reports 


©  2009  Lockheed  Martin  Corp. 


10/29/2009  11 


EDQP  Stages  of  Acknowledgement  ^ 

•  Candidate 

-  Interest  in  career  in  the  subject  discipline,  but  experience  or  skill  level 
requirements  for  for  Qualification  not  yet  met. 

-  Application  for  EDQP  of  the  subject  discipline  has  been  accepted. 

-  Formalizes  career  development  intent  and  planning. 

-  Pre-requisites  achieved  per  documented  requirements  (in  270-17). 

•  Qualified 

-  An  individual  who  has  met  or  exceeds  the  criteria  specified  for  the  Qualified 
Stage  in  the  specific  discipline. 

-  The  minimum  common  criteria  to  attain  the  designation  of  “Qualified”  is 
documented  for  each  discipline  in  the  appendices  of  270-17. 

-  The  business  unit  may  include  additional  criteria  (e.g.,  to  address  domain 
or  business  unit  specific  needs)  in  their  implementation  of  the  program. 

•  Advanced 

-  An  individual  who  has  met  or  exceeds  the  criteria  specified  for  the 
Advanced  stage  in  the  specific  discipline. 

-  The  minimum  common  criteria  to  attain  the  designation  of  “Advanced”  is 
documented  for  each  discipline  in  the  appendices  of  270-17. 

-  The  business  unit  may  include  additional  criteria  (e.g.,  to  address  domain 
or  business  unit  specific  needs)  in  their  implementation  of  the  program. 
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Notional  Development  Timeline 


Advanced  stage 


Ordered  progression 
through  the  stages  is 
expected  but  not  required 


Qualified  stage 


Candidate  stage 


Engineer 


\ 

o 


- 1 - 

No  Minimum  to 
Start 

(except  for 
Architects) 


— I — 

X 

Minimum 
Experience 
Req.  for 
Qualified 


— I — 

Y 

Minimum 
Experience 
Req.  for 
Advanced 


Amount  of  relevant  experience  for  each 
stage  varies  by  role.  Generally,  5-10 
years  for  Qualified  stage  and  an 
additional  3-5  years  for  advanced  stage 


Years  of  Relevant  Engineering  Experience 


Existing  employees  and  new  hires  can 
be  assessed  at  any  point  in  their  career 
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Other  Information  in  EDQP 

•  EDQP  Concept  of  Operations 

•  Eligibility 

-  Open  to  all,  except  where  pre-requisites  are  noted 

•  Successful  completion  of  training 

-  Testing  is  on  course-by-course  basis  per  learning  objectives 

•  Request  for  Acceptance  of  Equivalent  Learning  or  Development 

-  No  blanket  waivers  or  grandfathering 

-  Provide  rationale  for  equivalency  with  objective  evidence 

•  Reciprocity 

-  Accepted  by  receiving  BU 

-  Employee  responsible  to  obtain  domain  skills  per  BU  needs 

•  Renewal 

-  Business  Unit  decision 

-  Typically  3-5  years  with  additional  learning  and  experience 
requirements 


©  2009  Lockheed  Martin  Corp. 


10/29/2009  14 


Skill  Set  Matrix 

*  Documents  the  skills  required  for  given  disciplines  or 
roles 

*  Includes  skill  categories,  skill  sets,  skills,  sub-skills  and 
appropriate  classifications 

-  Skill  Category  -  High-level  grouping  of  skill  sets  based  on 
general  focus 

-  Skill  Set  -  A  set  of  skills  that  are  related  to  a  key  objective. 

-  Skill  -  Aptitude  required  for  the  performance  of  a  process  or  life 
cycle  activity. 

-  Sub-skill  -  One  of  lower  level  multiple  aptitudes  required  to 
perform  a  skill. 

*  Skill  Sets,  Skills,  and  Subskills  are  defined  the  discipline 
team  for  each  skill  category 


development 
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Common  Skill  Categories 

•  Process 

-  Common  skills  apply  to  all  disciplines 

-  Addresses  organizational  standard  processes,  standards, 
and  tools 

•  Technical 

-  Focused  on  the  technical  engineering  processes  through 
the  life  cycle 

•  Application/Domain/Environment  (BU  Specific) 

-  Skills  specific  to  the  business  unit  domain  areas 

•  Personal  Development 

-  Common  set  established  by  CTLC  for  all  disciplines 

-  Focused  on  the  interpersonal,  communication,  efficiency 
and  effectiveness,  and  team  skills 

•  Management 

-  Focused  on  the  project  management  processes  through 
the  life  cycle 
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Curriculum  Development 


Derived  from  the  skills  to  provide  the  education^ 
building  the  skills 

-  Skills-^Learning  Requirements^Coyj 
iterations) 

-  Maintain  a  mapping  of  skills  to  ensure 

adequate  coverage  and  sui|^MlERjWtt)pinalysis 

Courses  included  in  th^^noyjjif^^J|^wndent  of  any  existing 
course 

-  Defines  whal^P^JK^M9^»S§  to  meet  the  LMC  skill 
requirei 

BT  -A  A  \ 

to: 

_ —  v.iternal  and  external) 

fffements  for  development  of  new  courses 


^  ^  ^JTf3  r  I  r  I : 


IS  f  r  '1**11 


rfiCTum  is  based kq 


g 


urse  offerings 
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Course  Types 

•  Essential  (Foundation)  courses 

-  Technical  knowledge  in  a  discipline  needed  for  fundamental 
tasks. 

•  Enhancement  (Supplemental)  courses 

-  More  in-depth  technical  knowledge  needed  for  more  advanced 
tasks. 

•  Specialization  courses 

-  Technical  knowledge  in  required  only  for  specialized 
assignments  in  that  discipline. 

•  Inter-discipline  courses 

-  Address  skills  in  one  discipline  that  are  beneficial  for 
successful  performance  in  other  disciplines. 

•  Personal  Development  courses 

-  Address  skills  that  enhance  general  professional  effectiveness. 

•  Domain/BU  Specific  courses 

-  Defined  by  the  BU  to  meet  unique  needs 
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System  Engineer  and  Architect 
Development 


LMC  SE 
Development/ 
Qualification 

Program  (in  piac< 
in  many  BUs) 


The  same  approach 
is  being  developed 
for  SWE  &  SWA 


>5J/J! 


Career  Dev. 
Plan  -  based  on 
qualification 


LMC  SA 

Development/ 

Qualification 

Program 
Piloting) 
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Continuous  Improvement 


•  Alignment  with  SE  competency  models 

-  Influence,  learn  from  and  align  with  efforts 
across  industry  (e.g.,  NDIA,  UARC,  INCOSE) 

*  Refine/improve  over  time 

-  Monitor  changes  in  technology,  customer  needs, 
and  advancements  in  learning  approaches 

-  Incorporate  lessons  learned 
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QUESTIONS? 
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NDIA  12th  Annual  Systems  Engineering  Conference 
TheRole  of  Simulation  to  Track  Mobile  Assets  Using 
t„a  "  Automatic  Identification  Systems 


Swee  Leong 
NIST 

Swee.Leong@nist.gov 

301-975-5426 


October  26-29,  2009 


National  Institute  of  Standards  and  Technology 

Techno  fogy  Administration,  U.S.  Department  of  Commerce 


Innovation  &  productivity 


Introduction 


Boeing  is  collaborating  with  the  National 
Institute  of  Standards  and  Technology  (NIST)  to 
model  mobile  assets  for  777  and  787  final 
assembly  operations 

Evaluations  will  be  applied  to  assess  the 
business  case  in  the  use  of  auto  ID  technologies 


NIST  Core  Manufacturing  Simulation  Data 
(CMSD)  Information  Model 

Boeing  Material  Handling  System  discrete  event 
simulation  model 


Kill  Hypothetical  Case 


Asset  /Vehicle  /  Equipment 
Management 


Problem  Statement 


DARATECH,  Inc. 


t  cf  *  most  global  and  complex  companies  ate 
lb7e  to  foil,  exp, olt^-  gloM  ,««teofa,«. 

"PSs™Ou“^“ha'ttt-e^pe-fomaoces 

SO^tcent  lower  than  that  of  similarly  large  and  “"J' “ 

£5SS  with  the  capabilities  to  exploit  more  folly  the,  _ 


strv  update 

IT  for  MANUFACTURING,  ENGINEERING,  CONSTRUCTION  AND  PLANT  OPERATIONS 


no, 

S'r1*4**™* 


MUDDY  WATERS 

Lack  of  Interoperability  Continues 
to  Vex  Manufacturing  Industry; 
Diminishes  PLM  Spoils 

Proliferation  of  Standards  &  Data  Formats  Complicate  the  Issue 

By  Tim  Hickey 


never-before-realized  gains  in  manufactur- 
is  abuzz  with  ing  productivity:  fewer  physical  prototypes 
- hnilf  now  than  ever  before  —  in  the 

.a  _  -to  Failure 

Migration  proper  tan  = 


Over  95%  oJi-  Leading 

Standish  Group  SS°n  Ration - 

tow  a 


Lack  of  interoperability:  $1B/yr  to  U.S.  auto  suppliers 

$3.9B/yr  in  electronics 


„„  ,  . .  PILU|._.|J|Jr 

“uptea  integration  direa 
Occasionally.  pre-buiKaa 

Michael  LlACrr°date  Varie*y 


Hosw-.  cto,  Perva;;;: 

Plenary  -  Standards  Activity  Committee 


and 


Software 


Change  -  A  White  Paper” 


Motivation  /  Issues 


Industry 

•  Use  visualization  and  simulation  to  analyze  bottleneck  and  equipment 
downtime  to  increase  capacity  and  improve  throughput 

•  Engineers  spent  too  much  time  and  effort  to  prepare  and  process  input 
data  to  simulation 

•  Engineers  take  too  long  to  create  simulation  models 
NIST 

•  Mission:  help  industry  improves  productivity  and  competitiveness  with 
visualization,  modeling  and  simulation 

•  Validate  CMSD:  exchange  manufacturing  resource  data 

•  Require  systems  integration  to  address  interoperability  among 
manufacturing  applications 


Plenary  -  Standards  Activity  Committee 


A  CMSD  Pilot  Implementation 


Goal 


The  CMSD  Information  Model  defines  a  data  specification  for  efficient 

exchange  of  manufacturing  data  in  a  manufacturing  simulation  environment. 
The  specification  provides  a  neutral  data  format  for  integrating 
manufacturing  applications  and  simulation. 


•  Enable  data  exchange  between  manufacturing  simulation  systems,  other 
software  applications,  and  databases 

•  Support  the  construction  of  manufacturing  simulators 

•  Support  testing  and  evaluation  of  manufacturing  software 

•  Support  manufacturing  software  application  interoperability. 


Scope 


The  CMSD  Information  Model  describes  the  entities  in  the  manufacturing 
domain  and  the  relationship  between  these  entities  that  are  necessary  to 
create  manufacturing  simulations. 

Manufacturing  data  includes,  but  not  limited  to: 

-  Resource  information 

-  Part  and  Inventory  information 

-  Process  planning 

-  Production  operations 

No  specification  of  implementation  methods  and  execution  behavior  of 
manufacturing  system 


Major  Data  Categories 


Organization 

Calendar 

Schedule 

Work 

Process  plan 
Operation  definition 
Resource 
Skill  definition 
Setup  definition 


•  Part 

•  Bill-of-Materials 

•  Inventory 

•  Maintenance  plan 

•  Revision 

•  Probability  distribution 

•  Reference 


A  CMSD  Pilot  Implementation 


Plenary  -  Standards  Activity  Committee 


Pacer  Test  Data  Set 


^3  Microsoft  Excel  -  Input  data  [  —  |[  n1  ][ X  ]  ' 


:  ^  1  File  Edit  View  Insert  Format  Tools  Data  Window  Help  Type  a  question  for  help  -  _  fi1 

-  -  ;  i,  ^  ||  ^  ^  a  m  m  m  m  !  $  %  >  Tq8  -J8  iw  im  ej  -  - 

_ A2  _ ^  15232 _ 


> 

m 

c 

D 

E 

F 

G  I 

H 

1 

ID  Coi 

line 

CC 

Request  Time 

Response  Time 

Request  Message 

Response  Message 

2 

152321777 

774 

330 

2/15/2009  5:51 

2/15/2009  6:27 

Ln=774 ,CC=330 ,Com=BFE  -  Small  Parts ,LnPs=SI  Pos  #3,Jobs=  ,  Please  send  me  the  Galley  ch  I'm  soory  that  job  is  not  avl.  for  line  774 

3 

15233  777 

775 

330 

2/15/2009  7:38 

2/16/2009  6:07 

Ln=775 ,CC=330 ,Com=BFE  -  Galley, LnPs=  SI  Pos  #2  ,Jobs=  ,  Please  send  the  Door  1  Galley's  fl  f  Good  Morning  John.cm  We  are  working  on  your  request. 

i  4 

15233  777 

775 

330 

2/15/2009  7:38 

2/16/2009  6:54 

Ln=775 ,00=330  ,Com=BFE  -  Galley ,LnPs=SI  Pos  £2,Jobs=  ,  Please  send  the  Door  1  Galley's  fl  f 

Good  Morning  John.cm  Transportation  called  @6:32  AM. 

5 

15234  777 

773 

330 

2/15/2009  8:02 

2/15/2009  8:33 

Ln=773  ,CC=330  ,Com=Ceiling  ,LnPs=FBJ  ,Jobs=  IP-00DGU7000W,  Please  send  ceiling  for  GU100C 

at  planeside  2/1 5 

6 

15235  777 

775 

330 

2/15/2009  8:21 

2/15/2009  9:26 

Ln=775  ,CC=330  ,Com=St  owbin  ,LnPs=SI  Pos*2,Jobs=  IP-0000003455,  IP-0000033238,  IP-000003 

7  carts  FWD  &  AFT  centers,  1  cart -30  & -35D  closeouts 

7 

15236  777 

775 

330 

2/15/2009  8:22 

2/16/2009  6:08 

Ln=775 ,CC=330 ,Com=BFE  -  Galley ,LnPs=SI  Pos  #2,Jobs=  ,  please  send  me  the  FWD  center  BA 

Good  Morning  John.cm  We  are  working  on  your  request. 

rs~ 

9 

15236  777 

15237  Jar r 

775  330 
775  330 

2/15/2009  8:22 
2/15/2009  8:45 

2/16/2009  6:54 
2/15/2009  9:50 

Ln=775 ,CC=330 ,Com=BFE  -  Galley ,LnPs=SI  Pos  ri2,Jobs=  ,  please  send  me  the  FWD  center  BA 

Good  Morning  John.cm  Transportation  called  @6:32  AM. 

Ln=775, CC=330, Com=Jamco  -  Lavs,LnPs=SI  Pos  *2,Jobs=  ,  Please  send  LAV2A-1  L72A-2L,  LA\ 

tagged  and  called  2/15/09  10:00 

10 

15238  777 

772 

122 

2/16/2009  5:38 

2/16/2009  5:49 

Ln=772, CC=1 22, Cam-Ceiling  ,LnPs=F/A  40-25  Pos  1  ,Jobs=  IP-00DGU6201 W,  IP-00DGU6202W, 

l/W 

1 1 

15238  777 

772 

122 

2/16/2009  5:38 

2/16/2009  6:07 

Ln=772  ,CC=1 22  ,Com=Ceiling  ,LnPs=F/A  40-25  Pos  1  ,Jobs=  IP-00DGU6201 W,  IP-00DGU6202W, 

Delivered  1  cart  thru  transportation. 

1  12 

15238  777 

772 

122 

2/16/2009  5:38 

2/16/2009  13:29 

Ln=772, CC=122, Com=Ceiling  ,LnPs=F/A  40-25  Pos  1  ,Jobs=  IP-00DGU6201 W,  IP-00DGU6202W, 

correct  parts  received  at  planeside  2/16/09  a.m. 
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15239  777 

774 

330 

2/16/2009  6:01 

2/16/2009  6:25 

Ln=774 ,CC=330 ,Com=BFE  -  Small  Parts ,LnPs=SI  Pos  #3,Jobs=  ,  please  send  me  the  Galley  chi 

Line  774  does  not  call  out  for  job  14167  per  PQR340.  Soi 

14 

15240  Jarr 

772 

122 

2/16/2009  7:54 

2/16/2009  8:47 

Ln=772  ,CC=1 22  ,Com=Jamco  -  Lavs  ,LnPs=F/A  40-25  Pos  1  ,Jobs=  ,  please  send  3f-lc,ip  #dgt  61 C 

all  lavs  at  planeside  2/16 

is 

15241  Jar r 

15242  777 

772 

122 

2/16/2009  7:59 

2/16/2009  8:47 

Ln=772  ,CC=1 22  ,Com=Jamco  -  Lavs  ,LnPs=F/A  40-25  Pos  1  ,Jobs=  ,  please  send  the  other  2  lavs  : 

all  your  lavs  are  at  planeside  2/16 

16 

775 

140 

2/16/2009  8:30 

2/16/2009  8:53 

Ln=775  ,CC=1 40  ,Com=BFE  -  Doghouse  ,LnPs=40-24. 2 ,  J-2. 5  ,Jobs=  ,  1 030g53a00000  □□  1 030g55; 

Good  Morning  Steven, □  □  Will  pick  &.  pack  your  dogs  and 

17 

15243  777 

775 

140 

2/16/2009  8:33 

2/16/2009  9:02 

Ln=775 ,CC=1 40 ,Com=Adapter  Panels ,LnPs=40-24. 2 ,  J-2.5,Jobs=  ,  Send  all  140  panels  please 

They  are  ready  and  waiting  for  Trans,  to  be  shipped  over 

IS 

15243  777 

775 

140 

2/16/2009  8:33 

2/16/2009  9:46 

Ln=775 ,CC=1 40 ,Com=Adapter  Panels ,LnPs=40-24. 2 .  J-2.5,Jobs=  .  Send  all  140  panels  please 

Delivered  to  Steve  Evans  for  line  775  2/16/09 

19 

15244  777 

772 

122 

2/16/2009  9:15 

2/16/2009  9:37 

Ln=772  ,CC=1 22  ,Com=BFE  -  Small  Parts  ,LnPs=F/A  40-25  Pos  1  ,Jobs=  ,  Please  send  down  seat 

available  at  124w  pis  paper  work  thank  you. 

20 

15245  7^ 

772 

122 

2/1 6/2009  9:35 

2/16/2009  9:44 

Ln=772, 00=122, Com=Ceiling,LnPs=F/A  40-25  Pos  1  ,Jobs=  IP-0000004074,  IP-0000004075,  Plea 

I/W 

fit 

15245  777 

772 

;  122 

2/16/2009  9:35] 

2/16/2009  10:04 

Ln=772 ,CC=1 22 ,Com=Ceiling  ,LnPs=F/A  40-25  Pos  1  ,Jobs=  IP-0000004074,  IP-0000004075,  Plea 

HI  shipping  out  4  carts 

I  22 

15245  777 

772 

122 

2/16/2009  9:35 

2/16/2009  10:44 

Ln=772 ,CC=1 22 ,Com=Ceiling  ,LnPs=F/A  40-25  Pos  1  ,Jobs=  IP-0000004074,  IP-0000004075,  Plea  4  carts  at  planeside  2/16/09 

"23 

15246  777 

772 

330 

2/16/2009  14:45 

2/16/2009  15:33 

Ln=772 ,CC=330 ,Com=BFE  -  Galley  Loose  Parts  Kit  ,LnPs=F/A  40-25  Pos  1  ,Jobs=  IP-0000014211 

1  have  looked  around  and  see  nothing  left  forthis  a/p,  sorr; 

24 

15248  777 

775 

330 

2/16/2009  15:25 

2/16/2009  15:42 

Ln=775 ,CC=330 ,Com=Crewrest/OSU ,LnPs=SI  Pos  ri2,Jobs=  ,  Please  send  me  the  dgr8501w  and8503w?  Is  this  the  correct  IP?  Are  you  looking  for  the  bur 

25 

15248  777 

775  330 

2/16/2009  15:25 

2/16/2009  15:56 

Ln=775 ,00=330  ,Com=Crewrest/OSU  ,LnPs=SI  Pos  ri2,Jobs=  ,  Please' send  me  the  dgr8501w  and 

dgr8501w  ready  for  p/u  called  trans  at  3:50pm.  Thanks!! 

26 

15248  777 

775 

330 

2/16/2009  15:25 

2/16/2009  15:34 

Ln=775 ,CC=330 ,Com=Crewrest/OSU ,LnPs=SI  Pos  #2,Jobs=  ,  Please  send  me  the  dgr8501w  and 

in  work 

27 

15248  777 

775 

330 

2/16/2009  15:25 

2/16/2009  20:59 

Ln=775 ,CC=330 ,Com=Crewrest/OSU ,LnPs=SI  Pos  ri2,Jobs=  ,  Please  send  me  the  dgr8501w  and 

dgr  8501  @  pos#3 

28 

15249  777 

772 

330 

2/16/2009  16:21 

2/16/2009  21 :00 

Ln=772  ,CC=330  ,Com=Ceiling  ,LnPs=F/A  40-25  Pos  1  ,Jobs=  ,  We  need  the  ceiling  panels  for  IP-0C 

loc  @  pos  f/a  pos  #1 

29 

15249  777 

772 

330 

2/16/2009  16:21 

2/16/2009  16:33 

Ln=772  ,CC=330  ,Com=Ceiling  ,LnPs=F/A  40-25  Pos  1  ,Jobs=  ,  We  need  the  ceiling  panels  for  IP-0C 

in  work 

30 

15249  777 

772 

330 

2/16/2009  16:21 

2/16/2009  17:00 

Ln=772  ,CC=330  ,Com=Ceiling  ,LnPs=F/A  40-25  Pos  1  ,Jobs=  ,  We  need  the  ceiling  panels  for  IP-0C 

Hey  John -2  carts  ,  no  shorts  ,  at  40-56  trans.  aisle  .  Trar 

31 

15250 

777 

772 

122 

2/16/2009  17:59 

2/16/2009  18:24 

Ln=772  ,CC=1 22  ,Com=BFE  -  Small  Parts  ,LnPs=F/A  40-25  Pos  1  ,Jobs=  ,  Requesting  IP-00DGS8E 

1  have  no  l-P  in  BFE  with  this  number,  If  you  are  looking  fc 

32 

15251 

777 

775 

330 

2/16/2009  18:14 

2/16/2009  18:41 

Ln=775, CC=330, Com=Crewrest/OSU,LnPs=SI  Pos  *2,Jobs=  IP-00DGR8502W,  thanks 

In  work. 

33 

15251 

777 

775 

330 

2/16/2009  18:14 

2/16/2009  19:41 

Ln=775,CC=330,Com=Crewrest/OSU,LnPs=SI  Pos  *2,Jobs=  IP-00DGR8502W,  thanks 

Sent  five  units  449W51 1 0-45B  .-50B  ,-51  B  ,-52B  ,-53B.to  Ln^ 

34 

15251 

777 

775 

330 

2/16/2009  18:14 

2/16/2009  20:11  Ln=775 ,00=330 ,Com=Crewrest/OSU ,LnPs=SI  Pos  #2,Jobs=  IP-00DGR8502W,  thanks 

Notified  at  approximately  7:45  P.M.  that  the  Factory  is  □  [ 

35 

15251 

777 

775 

330 

2/16/2009  18:14 

2/16/2009  20:53  Ln=775 ,CC=330 ,Com=Crewrest/OSU ,LnPs=SI  Pos  *2,Jobs=  IP-00DGR8502W,  thanks 

loc  @  pos  #3 

36 

15251 

777 

775 

330 

2/16/2009  18:14 

2/16/2009  23:10 

Ln=775,CC=330,Com=Crewrest/OSU,LnPs=SI  Pos  #2,Jobs=  IP-00DGR3502W,  thanks 

Called  transportation  again,  will  be  delivered  after  1st  brea 

37 

15257 

777 

775 

330 

2/17/2009  5:14 

2/17/2009  5:27 

Ln=775 ,CC=330 ,Com=Cei ling  ,LnPs=SI  Pos#3,Jobs=  IP-0000041212,  IP-0000041213,  IP-0000041 

forwarding  to  first  shift 

33 

15257 

777 

775 

330 

2/17/2009  5:14 

2/17/2009  5:33 

Ln=775 ,00=330 ,Com=Cei ling  ,LnPs=SI  Pos#3,Jobs=  IP-0000041212,  IP-0000041213,  IP-0000041 

l/W 

39 

15257 

1/7 

775 

330 

2/17/2009  5 :  i~4 

2/1 7/2009  7:52^ 

Ln=775 ,00=330  ,Com=Cei  ling  ,LnPs=SI  Pos#3,Jobs=  IP-0000041212,  IP-0000041213,  IP-0000041 

Hi  Kenneth,  all  IP's  are  on  there  wy.  waiting  fortranspo.  tf 

40 

15258 

111 

773 

!  122 

2/17/2009  5:21 

2/17/2009  7:40 

Ln=773, 00=1 22, Com=  St owbin ,LnPs=F/A  40-25  Pos  1  ,Jobs=  IP-0000022034,  IP-0000022035,  IP-f 

At  planeside  2/17/09 

41 

15259  777 

773 

122 

2/17/2009  5:35 

2/17/2009  5:54 

Ln=773, 00=1 22, Com=  St owbin ,LnPs=F/A  40-25  Pos  1  ,Jobs=  IP-0000072551  .  IP-0000072552,  pie 

27  carts  in  all  includes  centers- □□□□  IP-0000022034 ,  IP-f 

42 

15260  777 

773 

122 

2/17/2009  5:54 

2/17/2009  6:03 

Ln=773, 00=1 22, Com=Sidewall,LnPs=F/A  40-25  Pos  1  ,Jobs=  IP-00DGW3207W,  IP-00DGW3208'. 

l/W 

43 

15260  777 

773 

122 

2/17/2009  5:54 

2/17/2009  6:08 

Ln=773, 00=1 22, Com=Sidewall ,LnPs=F/A  40-25  Pos  1  ,Jobs=  IP-00DGW3207W,  IP-00DGW3208'. 

1  am  notifying  Transportation...  ddl  am  shipping  3  sidewall 

44 

15260  777 

773 

122 

2/17/2009  5:54 

2/17/2009  7:48 

Ln=773, 00=1 22, Com=Sidewall,LnPs=F/A  40-25  Pos  1  ,Jobs=  IP-00DGW3207W,  IP-00DGW3208' At  planeside  2/17/09 

j  45 ' 

15261  777 

772 

122 

2/17/2009  6:42 

2/17/2009  7:31 

Ln=772, 00=122, Com=BFE  -  Small  Parts ,LnPs=F/A  40-25  Pos  2,Jobs=  IP-00DGA3451 W,  Please 

Avl  inside  124W,  pis  sign  for  parts 

46 

15262  777 

774 

330 

2/17/2009  6:42 

2/17/2009  6:56 

Ln=774 ,00=330  ,Com=OCAS  ,LnPs=SI  Pos  #3,Jobs=  ,  Please  send  me  the  OCAS  thank  you 

Good  Morning  John.cm  Transportation  called  @6:52  AM. 

47 

15263  777 

772 

122 

2/17/2009  7:48 

2/17/2009  8:18 

Ln=772, 00=122, Com=Ceiling,LnPs=F/A  40-25  Pos2,Jobs=  IP-00DGU6001 W,  Ceiling  pnl.  413w3- 

j7w 

48 

15263  777 

772 

122 

2/17/2009  7:48 

2/17/2009  8:34 

Ln=772, 00=122,  Com=Ceiling  ,LnPs=F/A  40-25  Pos2,Jobs=  IP-00DGU6001 W,  Ceiling  pnl.  413w3- 

Good  morning  James, 1  spoke  to  our  lead  in  the  shop.  The 

h  ►  h  \ Raw  data/  CMSD  structure  /  ]<  >  J 


j  Draw  -  |  Auto5hapes  -  "V  'X  □  O  Ml  C"  EH.  I  .  V?  T  T  t  =  =.=  5=C  -J  ^ 

Ready  NUM  5CRL 


•  The  Pacer  data  will  be  mapped  into  the  Core  Manufacturing  Simulation  Data  (CMSD) 
structure 


li  x 


Sorted  Pacer  Data 


C  Microsoft  Excel  -  Input  data  [-  ||  r?  |[X 


File  Edit  View  Insert  Format  lools  Data  Window  Help  Type  a  question  for  help  *  .  fl  X 


\  J  ^  A  a  -  y  _  &  X  'Mil  ja[g]o'  ^  \m* _ -n  -  b  i  u  m_  m  m  ~m  $  %  »  Tog  4°g ;  w  W  A- * 

A2  -  f*  15232 


A  T  B 

C 

D 

E 

F 

G 

H 

1 

J 

K 

L 

M 

N 

0 

P 

1  ID  line 

Request  Time 

Response  Time 

Delivery  Location 

Commodity 

2  15232  774 

2/15/2009  5:51 

2/15/2009  6:27 

SI  Pos  #3 

BFE  -  Small  Parts 

■clEErccIfiZ? 

2/15/2009  7:38 

2/16/2009  6:54 

SI  Pos  #2 

BFE-  Galley 

D 

15234 

773 

2/15/2009  8:02 

2/15/2009  8:33 

FBJ 

Ceiling 

5 

15235 

775 

2/15/2009  8:21 

2/15/2009  9:26 

SI  Pos  #2 

Stowbin 

6 

15236 

775 

2/15/2009  8:22 

2/16/2009  6:54 

SI  Pos  #2 

BFE  -  Galley 

7 

15237 

775 

2/15/2009  8:45 

2/15/2009  9:50 

SI  Pos  #2 

Jamco  -  Lavs 

8 

15238 

772 

2/16/2009  5:38 

2/16/2009  13:29 

F/A  40-25  Pos  1 

Ceiling 

9 

15239 

774 

2/16/2009  6:01 

2/16/2009  6:25 

SI  Pos  #3 

BFE  -  Small  Parts 

10 

15240 

772 

2/16/2009  7:54 

2/16/2009  8:47 

F/A  40-25  Pos  1 

Jamco  -  Lavs 

11 

15241 

772 

2/16/2009  7:59 

2/16/2009  8:47 

F/A  40-25  Pos  1 

Jamco  -  Lavs 

12 

15242 

775 

2/16/2009  8:30 

2/16/2009  8:58 

40-24.2,  J-2.5 

BFE  -  Doghouse 

13 

15243 

775 

2/16/2009  8:33 

2/16/2009  9:46 

40-24.2,  J-2.5 

Adapter  Panels 

14 

15244 

772 

2/16/2009  9:15 

2/16/2009  9:37 

F/A  40-25  Pos  1 

BFE  -  Small  Parts 

15 

15245 

772 

2/16/2009  9:35 

2/16/2009  10:44 

F/A  40-25  Pos  1 

Ceiling 

16 

15246 

772 

2/16/2009  14:45 

2/16/2009  15:33 

F/A  40-25  Pos  1 

BFE  -  Galley  Loose  Parts  Kit 

17 

15248 

775 

2/16/2009  15:25 

2/16/2009  20:59 

SI  Pos  #2 

Crewrest/OSU 

18 

15249 

772 

2/16/2009  16:21 

2/16/2009  17:00 

F/A  40-25  Pos  1 

Ceiling 

19 

15250 

772 

2/16/2009  17:59 

2/16/2009  18:24 

F/A  40-25  Pos  1 

BFE  -  Small  Parts 

20 

15251 

775 

2/16/2009  18:14 

2/16/2009  23:10 

SI  Pos  #2 

Crewrest/OSU 

21 

15257 

775 

2/17/2009  5:14 

2/17/2009  7:52 

SI  Pos  #3 

Ceiling 

22 

15258 

773 

2/17/2009  5:21 

2/17/2009  7:40 

F/A  40-25  Pos  1 

Stowbin 

23 

15259 

773 

2/17/2009  5:35 

2/17/2009  5:54 

F/A  40-25  Pos  1 

Stowbin 

24 

15260 

773 

2/17/2009  5:54 

2/17/2009  7:48 

F/A  40-25  Pos  1 

Sidewall 

25 

15261 

772 

2/17/2009  6:42 

2/17/2009  7:31 

F/A  40-25  Pos  2 

BFE  -  Small  Parts 

26 

15262 

774 

2/17/2009  6:42 

2/17/2009  6:56 

SI  Pos  #3 

OCAS 

27 

15263 

772 

2/17/2009  7:48 

2/17/2009  14:05 

F/A  40-25  Pos  2 

Ceiling 

28 

15264 

773 

2/17/2009  8:43 

2/17/2009  8:58 

F/A  40-25  Pos  1 

Ceiling 

29 

15265 

772 

2/17/2009  9:23 

2/17/2009  11:16 

F/A  40-25  Pos  2 

Carpets 

30 

15266 

773 

2/17/2009  9:35 

2/17/2009  9:44 

F/A  40-25  Pos  1 

Door  Liner 

31 

15267 

773 

2/17/2009  9:41 

2/17/2009  11:12 

F/A  40-25  Pos  1 

Sidewall 

32 

15268 

773 

2/17/2009  10:43 

2/18/2009  5:51 

F/A  40-25  Pos  1 

BFE  -  Galley 

33 

15269 

772 

2/17/2009  10:46 

2/17/2009  11:26 

F/A  40-25  Pos  1 

Carpets 

34 

15270 

775 

2/17/2009  13:33 

2/17/2009  14:31 

SI  Pos  m 

BFE  -  Small  Parts 

35 

15271 

775 

2/17/2009  13:43 

2/18/2009  5:18 

SI  Pos  #3 

OCAS 

36 

15272 

772 

2/17/2009  15:05 

2/17/2009  15:11 

F/A  40-25  Pos  2 

Mats 

37 

15273 

774 

2/17/2009  15:09 

2/17/2009  16:04 

FBJ 

BFE  -  Small  Parts 

38 

15274 

774 

2/17/2009  15:12 

2/18/2009  7:02 

FBJ 

Ceiling 

39 

15275 

772 

2/17/2009  15:15 

2/18/2009  5:57 

F/A  40-24 

Adapter  Panels 

413 

15276 

772 

2/17/2009  15:17 

2/17/2009  16:35 

F/A  40-24 

BFE  -  Small  Parts 

41 

15277 

773 

2/17/2009  15:38 

2/17/2009  16:36 

F/A  40-25  Pos  1 

BFE  -  Small  Parts 

IT 

15278 

772 

2/17/2009  15:45 

2/17/2009  16:36 

F/A  40-25  Pos  1 

BFE  -  Small  Parts 

43 

15279 

772 

2/17/2009  16:21 

2/17/2009  17:36 

F/A  40-25  Pos  2 

Mats 

'  44 

15281 

772 

2/17/2009  17:18 

2/17/2009  18:14 

F/A  40-25  Pos  2 

IRC  -  All 

45 

15282 

772 

2/17/2009  19:11 

2/17/2009  20:56 

F/A  40-25  Pos  2 

Partition 

46 

15283 

775 

2/17/2009  19:27 

2/17/2009  20:18 

SI  Pos  m 

BFE  -  Small  Parts 

47 

15284 

775 

2/17/2009  21:30 

2/17/2009  22:11 

SI  Pos  #3 

Carpets 

48 

15285 

775 

2/18/2009  1:56 

2/18/2009  2:43 

SI  Pos  #1 

Carpets 

|i<  <  ►  w  \  Raw  data  \CMSD  structure/  |<  >  | 


j  Draw  -  i£  AutoShapes  -  \  V  □  Q  jj  4  y  l&l  til  &  "  A  "  A  "  =  ^  ^ 

Ready  _  _ NUM  SCRL 


The  Pacer  test  data  has  been  sorted,  edited,  and  ready  to  be  mapped  to  the  CMSD 
data  structures 


Sample  CMSD  XML 


O  C:\Data\CMSD  data. xml  -  Windows  Internet  Explorer 

nwxi; 

^  V'-  -  |  jj|rj  C:\Data\CMSD  data.xml 

v  *t  X  Google 

ll*E) 

€E 

Ok  &  '£  C :  \Data\CMSD  data .  xml 

a  -  a  # 

’  Page  -  $  Tools  - 

-  <CMSDDocument> 

-  <DataSection> 

-  <Job> 

<Identifier>15232<^Identifier> 

<Status>started</Status> 

-  <ActualEffort> 

<StartTime >2/ 15/2009  5:51:55  AM</StartTime> 

<EndTime>2/15/2009  6:27:57  AM</EndTime> 

-  <Property> 

<Name  >Line  </Name  > 

<Value  >774 <^Value  > 

</Property> 

-  <Property> 

<Name  >DeliveryLocation  </Name  > 

<Value>SI  Pos  #3 Rvalue > 

</Property> 

-  <Property> 

<Name>Commodity</Name> 

<Value>BFE  -  Small  Parts</Value> 

</Property> 

</ActualEffort> 

</3ob> 

-  <Job> 

<Identifier>15233</Identifier> 

<Status>started</Status> 

-  <ActualEffort> 

<StartTime>2/ 15/2009  7:38:27  AM</StartTime> 

<EndTime>2/16/2009  6:54:11  AM<^EndTime> 

-  <Property> 

<Name  >Line  </Name  > 
cValue  >775  </Value  > 

</Property> 

-  <Property> 

<Name  ;  Delivery  Location  </Name  > 

<Value>SI  Pos  #2</Value> 

^Property  > 

-  <Property> 

<Name>Commodity</Name> 

<Value>BFE  -  Galley</Value> 

</Property> 

</ActualEffort> 

</3ob> 

+  <Job> 

+  <Job> 

+  <3ob> 

+  <Job> 

+  <Job> 

+  <Job> 

+  <-loh> 

J  My  Computer 

+V 100%  - 

•  The  first  row  of  the  Excel  file  mapped  into  the  CMSD  structure 


A  Hypothetical  Case  Simulation 


Data  for  all  delivery  stations 

To^al  Q'de'5  Handled 
Awdge  □  irJeis  pc>  'iVeet. 

^laeULage  cf  PinDlem  Oideis 
Mum  Dei  cf  PinDlem  Qideis 
Current  Order  HhAtA 
OammrrfLy  name 

Une  numDei 
Defve'f  kaoijnn 


v* 


•  q>^  ^  *4 


Line  772 

24 

3 

27 

lllllllllllllllllllllllllll 

A. 

Line_773 

31 

3 

34 

llllllllllllllllllllllllllllllllll 

Line  774 

27 

0 

27 

lllllllllllllllllllllllllll 

Line  775 

90 

4 

94 

mmmmmmmmmmmmmmmmmmmmmmmmmmmmmillllll 

Line 776 

77 

5 

32 

mmmmmmmmmmmmmmmmmmmmmmmmmillllll 

Line  777 

107 

6 

113 

llllllllllllllllllllllllllllllllllllllll^ 

mini 

Line  778 

92 

2 

94 

mmmmmmmmmmmmmmmmmmmmmmmmmmmmmillllll 

Lb - 

QZL. 

- Zl- 

immmmmmmmmmmmmmmmmmii 

1  TT 

Enter  Percentage  for  Problem  Orders  [X 

iiiiii 

i  sl 

i 

UN 

-1 

ij 

•  Develop  a  front  end  for  manufacturing  engineer  to  perform  what-if  scenarios  and 
iterations  of  simulation  and  analysis 


Sample  Arena  output 


j^j^jena  -  [Boeing  Simulation  model  ver66  -  Run  Mode] 


j  File  Edit  View  Tools  Arrange  Object  Run  Window  Help 


Hilf  dp 

-  a  xJr  xg>  .Cr 


□  c&  X  %  @  «  o.  |  g  p|3ix  3  ^  |  4.  %.  |  <y  g  a  |  j*J  m  ►►  11  m  ■ 


”  *? 


Advanced  Transfer 


Advanced  Process 


O  Flow  Process 


O  Reports 


“is  Navigate 


© 


=  m 

S-5  * 


Top-Level 

Y  Animation 

Y  ManagerLogic 

Y  Overall  view 

Y  StationA 

Y  StationB 

Y  StationC 

Y  StationD 

Y  StationE 

Y  StationF 

Y  StationLogic 


A  -  =  ▼  jnnn:  ▼  ; 


Total  Orders  Handled 

14  1  9  .  0  0  1 

Average  Orders  per  Week 

U  3  9  .  6  7  I 

Percentage  of  Problem  Orders 

|10  | 

Number  of  Problem  Orders 

|4  3  '  0  <n 

Current  order  Data 

Commodity  name 

floor  Mounted ...  I 

Line  number 

w  ..  1 

Delivery  location 

F7A  40-25  Pos  1  1 

Data  for  all  delivery  stations 


Number  of  Current  Orders 


sD*| 


Everett  Site  Map 


t 

N 


40-60  □ 


Legend 


(□ 


5 


*51 


S3 


fsT 


5 

27 

lllllllllllllllllllllllllll 

6 

34j 

llllllllllllllllllllllllllllllllll 

3 

27 

lllllllllllllllllllllllllll 

7 

94 

IIIIIIIIIIIIIIIIIIIIIIU 

6 

82 

lllllllllllllllllllllllllllllllllllllllllllllliy 

13 

113 

llllllllllllllllllllllllllllllllllllllll^ 

9 

95 

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiy 

10 

114 

IIIIIIIIIIIIIIIIIIM 

3 

45 

lllllllllllllllllllllllllllllllllllllllllllll 

7 

92 

llllllllllllllllllllllllllllllllllllllll^ 

4 

44j 

llllllllllllllllllllllllllllllllllllllllllll 

11 

110 

llllllllllllllllllllllllllllllllllllllll^ 

9 

88 

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiy 

2 

34  j 

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii^ 

7 

102 

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiy 

5 

33 

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii^ 

7 

41 

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 

5 

35l 

iiiiiiiiiiiiiiiiiim 

4 

5_7] 

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiy 

2 

33 

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 

0 

12 

iiiiiiiiiiii 

|For  Help,  press  FI 


1  /  1  (432.6415  Hours)  Sunday.  July  26,  2009 


Arena  animation  and  bar  chart  for  the  total  number  of  delivery  orders  processed. 


Sample  Arena  output 


Total  number  of  orders  per  line  number  in  Microsoft  Excel 


Value  Stream  Mapping  to  Quest  Model 


B  CM5QLjyeuiDAU.vid  •  -Microsoft  Yteie 


tilew 


\m&  >?  *  rd  g mi e  n aMa* .gnu  jg 


dl  *  -S'  r  T  - 


U  CtAT  _^S  H_  U  t»tfV4 


u&n9liVL' 


*aio| 

Afljfa  ] 

AOTO] 

AMD  | 

r-_ 

Pi,GH:e*j=i  1 

— 
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Value  Stream  Mapping  process  data  to  a  basic 

Delmia  Quest  Model 


Industry 

•  Use  visualization  and  simulation  to  meet  the  DoD’s  Manufacturing 
Readiness  Level  (MRL) :  Value  Stream  Map  (VSM)  process  data  and 
simulation  to  demonstrate  manufacturing  readiness. 

•  Engineers  spent  too  much  time  and  effort  to  prepare  and  process  input 
data  to  simulation 

•  Engineers  take  too  long  to  create  simulation  models 


Automatically  create  a  basic  Delmia  Quest  Model  from  Value  Stream 
Mapping  (VSM)  process  data. 


Supply  Chain  Simulation 


Sample  CMSD  interface  to  Arena 


Microsoft  Visual 


Test  Simply  CMSD  3  -  [ThisDocument  (Code)] 


File  Edit  View  Insert  Format  Debug  Run  Tools  Add-Ins  Window  Help 


i=H  T  (SI 

nn  i=M  rn 


[=l---ife^£  Project  (Test  Simply  rM' 

Arena  Objects 
|  Logic 

ThisDocument 


1 1  VBA_Block_l  EventBIC 

Alphabetic  j  Categorized  | 


nSf*  ^3?  m  Ln  1..  Col  : 
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Pic  A  vat  a  Smlzi  VB  A.  E  lock_l _ E  ire  (  ) 
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Dim  s  As  S  IHAN  1  Use  to  Joe  able  to  conmiunicate  with  Arena 

D  Am  >:m  1  £  A  1  a  nama  As  Stringy 


sponds  to  E  oolmmn  in  E  sc  o  a  i 


'  Tiro  var  Aalolas  to  write  to  Arena,  o  o  r  r  t 

Dim  :<  1  As  S  tr  l  n  g 

Dim  >:  2  As  String 

Dim  >:  3  As  String 

D  Am  ’-r  As  String 

D  Am  S  As  String 

D  Am  >:  >5  As  String 

Dim  V  As  String 

'  Tire  var  Aalolas  used  to  step  through  the  3£ I-I L  cl  o  o  mm  a  n  t 

D  im  m  yr  cl  o  o  As  New  D  OMD  o  c;  mma  nt  S  O 
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1 1  am  ( 2  ) 
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3 . 1 1  am  ( O  j 
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•  Sample  VBA  script  in  Arena 

•  Using  the  XML  DOM  standard  to  read  in  the  CMSD  XML  file 

•  Use  Arena  SIMAN  code  to  set  the  variables  in  Arena  model 


Sample  CMSD  UML  Diagram 


Sample  CMSD  UML  Diagram 


Sample  Arena  output 
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•  Arena  animation  and  bar  chart  for  the  total  number  of  delivery  orders  processed. 
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Purpose 


To  describe  how  the  DoD  Acquisition  Environment,  Safety,  and 
Occupational  Health  (ESOH)  Risk  Management  (RM)  process 
can  work  most  effectively  as  part  of  the  Systems  Engineering 
process 

To  highlight  common  elements  of  unsuccessful  and  successful 
ESOH  RM  processes 


Background 


Many  DoD  Acquisition  Program  Offices  have  tried  and  not  been 
very  successful  at  implementing  effective  and  efficient  ESOH 
RM  efforts,  while  some  Program  Offices  have  implemented 
programs  have  been  successful 

Based  on  lessons  learned  from  multiple  program  office 
experiences,  there  are  some  common  elements  of 
unsuccessful  and  successful  ESOH  RM  efforts 

How  do  you  feel  cibout  your  ESOH  RM  efforts?  hx 

▼ 

Unsuccessful  ESOH  RM  efforts  Successful  ESOH  RM  efforts 


USD(AT&L)  Policy  Memorandums  Related  to  ESOH 


Defense  Acquisition  System  Safety,  September  23,  2004 

-  Use  Standard  Practice  for  System  Safety,  MIL-STD-882D  to  manage  ESOH  risk 

-  Report  ESOH  risk  status  and  acceptance  decisions  at  technical  and  program 
reviews 

Reducing  Preventable  Accidents,  November  21, 2006 

-  Address  status  of  each  High  and  Serious  ESOH  risk  and  compliance  with 
applicable  safety  technology  requirements  at  all  program  reviews 

Defense  Acquisition  System  Safety  -  ESOH  Risk  Acceptance, 

March  7,  2007 

-  Formal  acceptance  of  ESOH  risks  prior  to  exposing  people,  equipment,  or  the 
environment  to  a  known  system-related  ESOH  hazard 

-  User  Representative  Formal  Concurrence  for  High  and  Serious  ESOH  risks 


These  basic  requirements  have  been  in  place  for  several 
years  &  incorporated  into  the  new  DoDI  5000.02 


Requirements 


•  December  8,  2008  DoD  Instruction  (DoDI)  5000.02  defines 
the  basic  requirements  for  Acquisition  Program  Office  ESOH 
RM  to  be  part  of  the  overall  Systems  Engineering  process 


4  TECHNICAL  REVIEWS  Tec 
conducted  when  the  svstem  mi.Hgr 
in  the  SEP  They  shall  include  pa 
the  program  (i.e..  peer  review),  un 
documented  a  the  SE? 

5  CONFIGURATION-  MANAGE 

approach  to  establish  ana  control  j 
system  life  cycle  This  approach ; 
physical  characteristics  of  the  syst 


_ _ _  H  Evaluation  iPESHEi.  The  PM  for  all  ar 

ACAT  level,  sb  ' - 

includes  the  following:  idiB 

ESOH  considerations  into  the  system^fi^^nzprocess.  identification  of  ESOH  risks  and 
a  description  of  the  method  for  ttacitlffl^^^hioughout  the  life  cycle  of  the 
system,  identification  of  hazardous  materials,  wastes,  andp^^^^iisc  barges  emissions 
noise)  associated  with  the  system  and  plans  for  their  minimization  anWhaj^yapsal.  and  a 
compliance  schedule  covering  all  system-related  activities  for  the  NEPA  (sec no 
ntle  42  of  U.S.C  (Reference  (ac)))  and  E.0. 12114  .(Reference  (ad))  The  Acquisition  Stra 
shall  incorporate  a  summary  of  the  PESHE.  including  die  NEPA  E.0. 12114  compliance 


b  NEPAEO  12114  The  PM  shall  conduct  and  document  NEPA  E  O  12114  analyses  for 
which  the  PM  is  the  action  proponent  The  PM  shah  provide  system-specific  analyses  and  data 
to  support  other  organizanons'  NEPA  and  E  0. 121 14  analyses.  The  CAE  (or  for  joint 
programs,  the  CAE  of  the  Lead  Executive  Component)  or  designee,  is  the  approval  authority  for 
system-related  NEPA  and  E.0  12114  documentation 

c.  Mishap  Investigation  Import  PMs  will  support  system-related  Class  A  and  B  mishap 
investigations  by  providing  analyses  of  hazards  that  contributed  to  the  mishap  and 
recommendations  for  matenel  risk  mitigation  measures,  especially  those  that  minimize  human 


7.  CORROSION  PREVENTION  AND  CONTROL.  As  pan  of  a  long-tetm  DoD  cc 
prevention  and  control  strategy  that  supports  reduction  of  total  cost  of  system  ownership,  each 
ACAT  I  program  shah  document  its  strategy  in  a  Corrosion  Prevention  Conirol  Plan.  The  Plan 
shall  be  required  at  Milestones  B  and  C.  Corrosion  considerations  shall  be  objectively  evaluated 


The  PM  shall  integrate  ESOH  risk 
management  into  the  overall  systems 
engineering  process  for  all 
developmental  and  sustaining 
engineering  activities.  As  part  of  risk 
reduction,  the  PM  shall  eliminate  ESOH 
hazards  where  possible,  and  manage 
ESOH  risks  where  hazards  cannot  be 
eliminated.  The  PM  shall  use  the 
methodology  in  MIL-STD-882D,  “DoD 
Standard  Practice  for  System  Safety”. 

DoDI  5000.02,  Enclosure  12 


Booz  Allen  Hamilton 


ENCLOSURE  12 


Guidance  for  ESOH  /  SE  Integration 


•  DoD  Defense  Acquisition  Guidebook  (DAG) 

-  Provides  detailed  guidance  on  how  DoD  expects 
Acquisition  Program  Offices  to  meet  the  ESOH  RM 
requirements  defined  in  DoDI  5000.02 

-  https://acc.dau.mil/dag 


v 


Welcome  i 


i  Interim  Defense  Acquisition  Guidebook  Homepage 


•  ESOH  In  Acquisition  -  Integrating  ESOH 
into  Systems  Engineering 

-  Depicts  when  ESOH  activities  should  be  performed  to 
influence  system  design  throughout  SE  process 

-  Includes  the  ESOH  Management  Evaluation  Criteria 
published  by  DDR&E  and  DUSD(I&E) 


•  Acquisition  Community  Connection  (ACC) 

-  Provides  best  practices  on  how  to  integrate  ESOH 
considerations  into  the  systems  engineering  and 
acquisition  processes 

-  https://acc.dau.mil/esoh 


Common  Elements  of  Unsuccessful  ESOH  RM  Efforts 


•  Disconnect  between  ESOH  Analysis  and  Design  Activities 

-  Difficult  to  implement  ESOH  recommendations  for  completed  SE  work  products 

-  ESOH  recommendations  will  meet  resistance  and  typically  have  limited  success 

-  Failure  to  follow  through  on  recommendations  and  to  work  to  viable  mitigation 
solutions  with  Design  Activities  and  the  User  Community 

-  Failure  of  E,  S,  and  OH  Subject  Matter  Experts  to  work  closely  together  with  SE 

»  E,  S,  and  OH  provide  conflicting  program  recommendations  on  same  issues 
»  SSWG  focused  only  on  Safety;  EWG  focused  only  on  Pollution  Prevention 

-  Failure  to  have  E  &  OH  Representatives  as  part  of  the  ESOH  effort 

-  Trying  to  communicate  a  major  design  change  to  reduce  ESOH  risk  at  the  wrong 
time  could  cost  the  program  significant  schedule  and  budget  -  obviously  this  will 
not  be  well-received 


Booz  Allen  Hamilton 
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Common  Elements  of  Unsuccessful  ESOH  RM  Efforts  (cont) 


ESOH  Personnel  are  viewed  by  Management  and  Engineering 
as  road  blocks,  not  team  members 

While  the  amount  of  resources  applied  to  the  ESOH  RM  efforts 
will  have  an  impact  on  the  quality  of  the  outcomes,  it  is  not  the 
most  critical  factor 

Many  large  Acquisition  Programs  have  allocated  significant 
resources  (funding  and  personnel)  for  ESOH  RM 

-  Can  produce  reduction  of  ESOH  risks  on  the  system  despite  organizational 
impediments 

-  For  example,  large  programs  have  been  doing  a  good  job  at  Hazardous 
Materials  Management 

-  However,  utilizing  substantial  program  funding  for  ESOH  RM  is  not  a  sustainable 
approach 


Common  Elements  of  Successful  ESOH  RM  Efforts 


•  An  ESOH  RM  effort  has  to  be  part  of  and  be  able  to  influence 
the  day-to-day  decision  making  that  occurs  in  the  Systems 
Engineering  process 

-  Provide  informative  and  timely  ESOH  inputs  to  Systems  Engineering 

»  Direct  line  of  communication  to  Systems  Engineering,  including 
Product/Engineering  Integrated  Product  Team  (IPTs) 

»  Daily  ESOH  communication  via  IPT  meetings,  phonecons,  test  logs 

»  Direct  line  of  communication  to  test  sites  and/or  end-users 

-  E,  S,  and  OH  Subject  Matter  Experts  work  together  to  optimize 
recommendations  across  these  functional  areas 

-  Implement  closed-loop  ESOH  hazard  tracking  system,  to  include  status  of 
recommended  mitigation  measures 

-  Integrate  ESOH  within  Configuration  Management  Processes 

»  ECR/ECP  reviews,  technical  document  reviews,  etc. 

»  Require  ESOH  review  and  approval  for  changes  to  be  finalized 

-  Participate  in  program  and  technical  reviews  (esp.  PDR  &  CDR)  to  report  risk 
and  applicable  ESOH  technology  status 


Booz  Allen  Hamilton 
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Common  Elements  of  Successful  ESOH  RM  Efforts  (cont) 


•  Program  Manager  and  Chief  Engineer  are  knowledgeable  and 
understanding  of  ESOH  efforts 

-  PM  and  Chief  Engineer  views  ESOH  as  team  members  and  not  as  roadblocks 

•  The  knowledge,  skills,  and  abilities  of  the  ESOH  practitioners 
supporting  a  program  can  have  a  significant  impact  on  the 
success  of  the  Acquisition  Program  Office's  ESOH  RM  efforts 

-  ESOH  practitioners  need  to  be  knowledgeable  in  their  system,  their  system’s 
operating  environment,  and  also  knowledgeable  in  applicable  laws  and 
regulations 

•  ESOH  Professionals  should  have  strong,  in-depth  knowledge 
of  the  ESOH  risks  AND  potential  mitigations 

-  During  IPT  meetings  and  before/during  design  reviews,  ESOH  participation  can 
provide  expert  feedback  real-time  to  best  influence  design  to  reduce  ESOH  risk 

-  During  test  site  visits  or  end-user  discussions,  ESOH  participation  can  receive 
real-time  feedback  on  suggestions  and/or  concerns  from  those  that  work  daily 
with  the  system  to  best  influence  design  to  reduce  ESOH  risk 


Booz  Allen  Hamilton 
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Common  Elements  of  Successful  ESOH  RM  Efforts  (cont) 


•  Programmatic  ESOH  Evaluation  (PESHE):  A  living  document  that  guides  and 
documents  identification  and  management  of  ESOH  risks. 

-  The  ONLY  DoD-required  ESOH  document! 

-  Successful  PESHEs  document  what  the  programs  plans  to  do  or  is  doing,  is  consistent  with  where  the 
program  is  in  the  life  cycle,  and  does  not  just  restate  policy 


big 

Starts  as  a  planning  document 


A  Start 

A  Initial  Completion 
A  Update  Required 


Becomes  an  ESOH  Risk  Management  Status 
Report 
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Conclusion 


•  If  the  ESOH  team  is  removed  from  the  Systems  Engineering 
process,  having  a  direct  line  to  the  Program  Manager  and/or 
having  a  large  ESOH  budget  may  not  effectively  influence 
design  changes  to  mitigate  ESOH  risk 

•  If  the  ESOH  RM  efforts  (resources  and  personnel)  are  a  fully 
integrated  part  of  the  Systems  Engineering  team  and  efforts, 
then  the  likelihood  of  having  a  successful  ESOH  RM  effort  will 
be  much  higher  than  a  better-resourced  ESOH  RM  effort  that  is 
operating  outside  of  the  System  Engineering  process,  even  if  it 
is  reporting  directly  to  the  Program  Manager 

•  Knowledgeable  and  experienced  ESOH  Professionals  can 
effectively  communicate  ESOH  risks  and  mitigations  on  a  day- 
to-day  basis  within  the  Systems  Engineering  process  to 
influence  design  changes  and  eliminate  or  reduce  risk 


Booz  Allen  Hamilton 
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Questions? 


Robert  E.  Smith,  CSP 
Booz  Allen  Hamilton 
1550  Crystal  Drive,  Suite  1100 
Arlington,  VA  22202-41 58 
703-412-7661 
smith_bob@bah.com 
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Acquisition  Life  Cycle 
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Fellow 
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pcroll@csc.com 

Chair,  NDIA  Software  Industry 
Experts  Panel 

Industry  Co-Chair,  NDIA 
Systems  Assurance  Committee 


P.  Croll 


12th  Annual  NDIA  Systems  Engineering  Conference,  29  October  2009 


1 


Outline 


■  Setting  the  Stage  for  Static  Code  Analysis 

-  What  is  Static  Code  Analysis? 

-  The  Scope  of  The  Problem 

-  Testing  vs.  Static  Code  Analysis 

-  What  Code  Do  You  Analyze? 

-  A  Three-Phase  Code  Analysis  Process 

-  The  Assurance  Case 

■  Static  Code  Analysis  in  the  Acquisition  Life 
Cycle 

■  Challenges  to  Effective  Static  Code  Analysis 

■  Useful  Links 


P.  Croll 


12th  Annual  NDIA  Systems  Engineering  Conference,  29  October  2009 


Setting  the  Stage  for  Static 

Code  Analysis 


P.  Croll 


12th  Annual  NDIA  Systems  Engineering  Conference,  29  October  2009 


What  is  Static  Code  Analysis? 


■  Static  code  analysis  is  the  process  of  evaluating 
a  system  or  component  based  on  its  form, 
structure,  content,  or  documentation.  From  a 
software  assurance  perspective,  static  analysis 
addresses  weaknesses  in  program  code  that 
might  lead  to  vulnerabilities 

■  Such  analysis  may  be  manual,  as  in  code 
inspections,  or  automated  through  the  use  of 
one  or  more  tools 

■  Automated  static  code  analyzers  typically  check 
source  code  but  there  is  a  smaller  set  of 
analyzers  that  check  byte  code  and  binary  code, 
especially  useful  when  source  code  in  not 
available  (e.g  for  COTS  components). 


P.  Croll 
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The  Scope  of  The  Problem 


Civilian 

Government 

Projects 

Military 

Projects 

Average 

Size  in  FP 

1 

1 

1 

1 

10 

5 

4 

5 

100 

29 

14 

24 

1,000 

155 

55 

120 

10,000 

832 

209 

600 

100,000 

4,467 

794 

3,031 

1.000,000  23,988  3,020  15,412 


Average  4,211  585  2,742 

Figure  1.  Estimated  Number  of  Security 
Vulnerabilities  in  Software  Applications.  Source: 
Capers  Jones  ©  2008 
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Figure  2.  Probability  of  Serious  Security 
Vulnerabilities  in  Software  Applications.  Source: 
Capers  Jones  ©  2008 


■  For  military  projects,  as  one  approaches  systems  the  size  of  typical  large  combat  systems  (expressed 
as  function  points),  the  estimated  number  of  security  vulnerabilities  rises  to  above  3000  and  the 
probability  of  serious  vulnerabilities  rises  to  over  45% 

■  The  statistics  are  much  worse  for  civilian  systems.  As  we  move  more  and  more  into  COTS  and  open 
source  software  for  our  combat  systems,  one  might  expect  that  the  true  extent  of  vulnerabilities  in  our 
systems  would  lie  somewhere  between  those  of  military  and  civilian  systems. 
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COTS  and  Open  Source 
Exacerbate  the  Problem 


■  Reifer  and  Bryant  [2]  studied  100  packages  were  selected  at  random  from  50 
public  Open-Source,  COTS,  and  GOTS  libraries 

-  Spanned  a  full  range  of  applications  and  sites  like  SourceForge 

-  Over  30%  of  Open  Source  and  GOTS  (Government  Off  the  Shelf)  packages 
analyzed  had  dead  code 

-  Over  20%  of  the  Open  Source,  COTS,  and  GOTS  packages  had  suspected 
malware 

-  Over  30%  of  the  COTS  packages  analyzed  had  behavioral  problems 

■  Reifer  and  Bryant  conclude  that  the  potential  for  malicious  code  in 
applications  software  is  large  as  more  and  more  packages  are  used  in 
developing  a  system. 


Figure  3.  COTS  Study  Findings.  Source:  D.  Reifer  and  E.  Bryant,  Software  Assurance  in 
COTS  and  Open  Source  Packages,  DHS  Software  Assurance  Forum,  October  2008 
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DoD  Clarifying  Guidance  Regarding  Open 
Source  Software  (OSS)  -  October  16,  2009 


2.  GUIDANCE 

a.  In  almost  all  cases,  OSS  meets  the  definition  of  “commercial  computer  software” 
and  shall  be  given  appropriate  statutory  preference  in  accordance  with  10  USC  2377 
(reference  (b))  (see  also  FAR  2.101(b),  12.000, 12.101  (reference  (c));  and  DFARS 

212.212,  and  252.227-701 4(a)(1)  (reference  (d))). 

c.  DoD  Instruction  8500.2,  “Information  Assurance  (IA)  Implementation,”  (reference 
(g))  includes  an  Information  Assurance  Control,  “DCPD-1  Public  Domain  Software 
Controls,”  which  limits  the  use  of  “binary  or  machine-executable  public  domain  software 
or  other  software  products  with  limited  or  no  warranty,”  on  the  grounds  that  these  items 
are  difficult  or  impossible  to  review,  repair,  or  extend,  given  that  the  Government  does 
not  have  access  to  the  original  source  code  and  there  is  no  owner  who  could  make  such 
repairs  on  behalf  of  the  government.  This  control  should  not  be  interpreted  as  forbidding 
the  use  of  OSS,  as  the  source  code  is  available  for  review,  repair  and  extension  by  the 
government  and  its  contractors. 

d.  The  use  of  any  software  without  appropriate  maintenance  and  support  presents  an 

information  assurance  risk.  Before  approving  the  use  of  software  (including  OSS), 

system/program  managers,  and  ultimately  Designated  Approving  Authorities  (DAAs), 

must  ensure  that  the  plan  for  software  support  (e.g.,  commercial  or  Government  program 

office  support)  is  adequate  for  mission  need. 

Source:  DoD  Chief  Information  Officer  (CIO)  Memorandum,  “Clarifying  Guidance  Regarding 
Open  Source  Software  (OSS)  in  the  Department  of  Defense  (DoD),”  October  16,  2009 
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Testing  vs.  Static  Code 

Analysis 


■  Testing  requires  code  that  is  relatively  complete 

■  Static  analysis  can  be  performed  on  modules  or 
unfinished  code  [4] 

■  A  static  analysis  tool  is  a  program  written  to  analyze 
other  programs  for  flaws 

-  Such  analyzers  typically  check  source  code 

-  A  smaller  set  of  analyzers  can  check  byte  code  and 
binary  code 

■  Manual  analysis,  or  code  inspection,  can  be  very 
time-consuming,  and  inspection  teams  must  know 
what  security  vulnerabilities  look  like  in  order  to 
effectively  examine  the  code 

■  Static  analysis  tools  are  faster  and  don’t  require  the 
tool  operator  to  have  the  same  level  of  security 
expertise  as  a  code  inspector  [5] 
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What  Code  Do  You  Analyze? 


■  How  do  you  prioritize  a  code  review  effort  when 
you  have  thousands  of  lines  of  source  code,  and 
perhaps  object  code  to  review? 

■  From  a  software  assurance  perspective,  looking 
at  attack  surfaces  is  not  a  bad  place  to  start  [6] 

-  A  system’s  attack  surface  can  be  thought  of  as 
the  set  of  ways  in  which  an  adversary  can  enter 
the  system  and  potentially  cause  damage 

-  The  larger  the  attack  surface,  the  more  insecure 
the  system  [7] 

-  Higher  attack  surface  software  requires  deeper 
review  than  code  in  lower  attack  surface 
components. 
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Heuristics  For  Code  Review  -  1 


■  Howard  proposes  the  following  heuristics  as  an  aid  to 
determining  code  review  priority  [8]: 

-  Old  code 

■  Older  code  may  have  more  vulnerabilities  than  new  code  because 
newer  code  often  reflects  a  better  understanding  of  security  issues 

■  Code  considered  “legacy”  code  should  be  reviewed  in  depth. 

-  Code  that  runs  by  default 

■  Attackers  often  go  after  installed  code  that  runs  by  default 

■  Such  code  should  be  reviewed  earlier  and  deeper  than  code  that 
doesn’t  execute  by  default 

■  Code  running  by  default  increases  an  application’s  attack  surface 

-  Code  that  runs  in  elevated  context 

b  Code  that  runs  in  elevated  identities,  e.g.  root  in  *nix,  for  example, 
also  requires  earlier  and  deeper  review  because  code  identity  is 
another  component  of  attack  surface. 

-  Anonymously  accessible  code 

■  Code  that  anonymous  users  can  access  should  be  reviewed  in 
greater  depth  than  code  that  only  valid  users  and  administrators  can 
access. 
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Heuristics  For  Code  Review  -  2 


-  Code  listening  on  a  globally  accessible  network  interface 

■  Code  that  listens  by  default  on  a  network,  especially  uncontrolled  networks  like 
the  Internet,  is  open  to  substantial  risk  and  must  be  reviewed  in  depth  for 
security  vulnerabilities. 

-  Code  written  in  C/C++/assembly  language 

■  Because  these  languages  have  direct  access  to  memory,  buffer-manipulation 
vulnerabilities  within  the  code  can  lead  to  buffer  overflows,  which  often  lead  to 
malicious  code  execution 

■  Code  written  in  these  languages  should  be  analyzed  in  depth  for  buffer- 
overflow  vulnerabilities 

-  Code  with  a  history  of  vulnerabilities 

i  Code  that’s  had  a  number  past  security  vulnerabilities  should  be  suspect, 
unless  it  can  be  demonstrated  that  those  vulnerabilities  have  been 
effectively  removed. 

-  Code  that  handles  sensitive  data. 

■  Code  that  handles  sensitive  data  to  should  be  analyzed  to  ensure  that 
weaknesses  in  the  code  do  not  disclose  such  data  to  untrusted  users. 

-  Complex  code. 

i  Complex  code  has  a  higher  bug  probability,  is  more  difficult  to 
understand,  and  may  likely  have  more  security  vulnerabilities. 

-  Code  that  changes  frequently. 

i  Frequently  changing  code  often  results  in  new  bugs  being  introduced 

■  Not  all  of  these  bugs  will  be  security  vulnerabilities,  but  compared  with  a 
stable  set  of  code  that’s  updated  only  infrequently,  code  that  is  less 
stable  will  probably  have  more  vulnerabilities  in  it. 
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A  Three-Phase  Code  Analysis 
Process  -  Phase  1 


■  Howard  [6]  also  suggests  a  notional  three- 
phase  code  analysis  process  that  optimizes 
the  use  of  static  analysis  tools. 

-  Phase  1  -  Run  all  available  code-analysis 
tools 

■  Multiple  tools  should  be  used  to  offset  tool  biases 
and  minimize  false  positives  and  false  negatives 

■  Analysts  should  pay  attention  to  every  warning  or 
error 

-  Warnings  from  multiple  tools  may  indicate  that  the 
code  that  needs  closer  scrutiny  (e.g.  manual  analysis). 

■  Code  should  be  evaluated  early,  preferable  with 
each  build,  and  re-evaluated  at  every  milestone. 
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A  Three-Phase  Code  Analysis 
Process  -  Phase  2 


■  Phase  2  -  Look  for  common  vulnerability  patterns 

-  Analysts  should  make  sure  that  code  reviews  cover  the 
most  common  vulnerabilities  and  weaknesses,  such  as 
integer  arithmetic  issues,  buffer  overruns,  SQL  injection, 
and  cross-site  scripting  (XSS) 

-  Sources  for  such  common  vulnerabilities  and  weaknesses 
include  the  Common  Vulnerabilities  and  Exposures  (CVE) 
and  Common  Weaknesses  Enumeration  (CWE) 
databases,  maintained  by  the  MITRE  Corporation  and 
accessible  at:  http://cve.mitre.org/cve/  and 
http://cwe.mitre.org/ 

-  MITRE,  in  cooperation  with  the  SANS  Institute,  also 
maintain  a  list  of  the  “Top  25  Most  Dangerous 
Programming  Errors”  (http://cwe.mitre.org/top25/index.htmn 
that  can  lead  to  serious  vulnerabilities 

-  Static  code  analysis  tool  and  manual  techniques  should  at 
a  minimum,  address  these  Top  25 
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A  Three-Phase  Code  Analysis 
Process  -  Phase  3 


■  Phase  3  -  Dig  deep  into  risky  code 

-  Analysts  should  also  use  manual  analysis 
(e.g.  code  inspection)  to  more  thoroughly 
evaluate  any  risky  code  that  has  been 
identified  based  on  the  attack  surface,  or 
based  on  the  heuristics  on  Slides  9  and  10 

-  Such  code  review  should  start  at  the  entry 
point  for  each  module  under  review  and 
should  trace  data  flow  though  the  system, 
evaluating  the  data,  how  it’s  used,  and  if 
security  objectives  might  be  compromised 
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The  Assurance  Case 


b  An  Assurance  Case  is  a  set  of  structured  assurance  claims,  supported 
by  evidence  and  reasoning  that  demonstrates  how  assurance  needs 
have  been  satisfied  [9] 

-  It  shows  compliance  with  assurance  objectives 

-  It  provides  an  argument  for  the  safety  and  security  of  the  product  or  service. 

-  It  is  built,  collected,  and  maintained  throughout  the  life  cycle 

-  It  is  derived  from  multiple  sources 

■  The  Sub-parts  of  an  assurance  case  include: 

-  A  high  level  summary 

-  Justification  that  product  or  service  is  acceptably  safe,  secure,  or  dependable 

-  Rationale  for  claiming  a  specified  level  of  safety  and  security 

-  Conformance  with  relevant  standards  and  regulatory  requirements 

-  The  configuration  baseline 

-  Identified  hazards  and  threats  and  residual  risk  of  each  hazard  and  threat 

-  Operational  and  support  assumptions 

■  An  Assurance  Case  should  be  part  of  every  acquisition  in  which  there  is 
concern  for  IT  security 

-  Should  be  prepared  by  the  supplier 

-  Should  describe 

■  The  assurance-related  claims  for  the  software  being  delivered, 

■  The  arguments  backing  up  those  claims, 

■  The  hard  evidence  supporting  those  arguments 
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Static  Code  Analysis  in  the 
Acquisition  Life  Cycle 


P.  Croll 


12th  Annual  NDIA  Systems  Engineering  Conference,  29  October  2009 


System  Engineering  Technical 
Review  Process  (SETR) 


b  DoDI  5000.02,  Operation  of  the  Defense  Acquisition  System  [10], 
describes  the  System  Engineering  Technical  Review  (SETR) 
process  associated  with  the  system  acquisition  life  cycle. 
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•  Initial  Technical  Review  (ITR) 

•  Alternative  Systems  Review  (ASR) 

•  Systems  Requirements  Review  (SRR) 

•  System  Functional  Review  (SFR) 

•  Preliminary  Design  Review  (PDR) 

•  Critical  Design  Review  (CDR) 

•  Post-PDR  Assessment  (Post-PDRA) 


•  Post-CDR  Assessment  (PCDRA)  •  Technology  Readiness 

•  Test  Readiness  Review  (TRR)  Assessment  (TRA) 

•  System  Verification  Review  (SVR)  •  In-Service  Review  (ISR) 

•  Functional  Configuration  Audit  (FCA) 

•  Production  Readiness  Review  (PDR) 

•  Operational  Test  Readiness  Review  (OTRR) 

•  Physical  Configuration  Audit  (PCA) 


Figure  4.  SETR  Process 
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Software  Cl  Reviews 


■  The  reviews  typically  associated  with 
software  are  shown  below 
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Figure  5.  Software  Cl  Reviews 

Source:  PEO IWS  Technical  Review  Manual  (TRM),  December  2008 
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System  Requirements  Review 

(SRR)  Objectives 


■  The  SRR  helps  the  PM  understand 
the  scope  of  the  software  assurance 
landscape  (assurance  requirements, 
elements  to  be  protected,  the  threat 
environment)  in  which  context  static 
code  analysis  should  be  applied. 
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System  Requirements  Review 

(SRR)  Outcomes 


■  Establishment  of  the  System  Assurance  Case 

-  Specification  of  the  top-level  system  assurance  claims  that  address 
identified  threats  to  the  mission. 

-  Identification  of  the  approach  for  developing  the  system  assurance  case. 

■  Identification  of  all  critical  elements  to  be  protected 

-  Identification  of  all  relevant  system  assurance  threats  and  their  potential 
impact  on  critical  system  assets. 

-  Identification  of  high-level  potential  weaknesses  in  the  system 

-  Determination  and  derivation  of  system  assurance  requirements  (as  a 
subset  of  the  system  requirements). 

■  Test  and  Evaluation  Master  Plan  (TEMP)  addressing  system 
assurance 

-  Examine  the  TEMP  to  ensure  testing  processes  are  sufficient  for  system 
assurance.  This  may  include  planning  for  static  code  analysis. 

■  Support  and  Maintenance  Concepts 

-  Documentation  of  the  support  and  maintenance  concepts  including  a 
description  of  how  assurance  will  be  maintained. 

-  Description  of  what  static  code  analysis  tools  will  be  used  post 
deployment  and  how  and  when  they  will  be  applied 
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Preliminary  Design  Review 
(PDR)  Objectives 


■  The  PDR  is  a  multi-disciplined 
technical  review  to  ensure  that  the 
system  under  review  can  proceed 
into  detailed  design,  and  can  meet 
the  stated  performance  requirements 
within  cost  (program  budget), 
schedule  (program  schedule),  risk, 
and  specific  assurance  requirements 
and  constraints. 
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Preliminary  Design  Review 
(PDR)  Outcomes  -  1 


b  Information  security  technology  evaluation  of  all  critical 
COTS/GOTS  elements 

-  Performed  as  part  of  the  analysis  of  alternatives. 

-  Includes  an  updated  assurance  case  based  on  the  design,  and  new 
weaknesses  and  vulnerabilities  identified. 

-  Results  of  static  code  analyses  performed  of  GOTS/COTS  components. 

■  Which  tools  were  used? 

■  What  weaknesses  and  vulnerabilities  were  discovered 

■  Specification  of  assurance-specific  static  analysis 

-  Specification  of  assurance-specific  static  analysis  and  assurance- 
specific  criteria  to  be  examined  during  code  reviews 

■  Code  reviews  performed  during  implementation 

■  Documented  in  the  System  Engineering  Plan  (SEP)  and  Software 
Development  Plan  (SDP) 

s  Plan  for  training  to  use  static  analysis  tools  and  for  manual  analysis 

■  Configuration  management 

-  For  Assurance,  the  preliminary  configuration  management  plan  must 
support  traceability  and  protection  of  each  configuration  item,  including 
requirements  and  architectural  elements. 

■  At  what  stages  of  the  configuration  management  process  will  static  code 
analysis  be  applied? 

■  What  configuration  change  events  will  trigger  code  analysis? 

■  What  components  will  be  analyzed? 

|  ■  How  will  the  results  of  the  analyses  be  documented? 
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Preliminary  Design  Review 
(PDR)  Outcomes  -  2 


■  Supply  Chain  Assurance 

-  For  all  critical  elements  being  considered  for 
procurement,  an  analysis  of  the  supplier  and  its 
processes  should  be  performed. 

b  Will  the  supplier  perform  static  code  analysis  as  part 
of  its  code  development  and/or  code  integration 
processes? 

■  Which  components  will  be  analyzed?  Which  will  not? 

b  What  tools  do  they  plan  to  use? 

b  What  are  the  details  of  their  code  inspection  process 
for  manual  security  analysis? 

b  How  will  they  mitigated  any  discovered  vulnerabilities 
or  weaknesses? 

■  Assurance  Case 

-  Updating  of  the  assurance  case  with  relevant 
evidence 
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Additional  Preliminary  Design 
Review  (PDR)  Considerations 


■  COTS  source  code  is  rarely  available  to  the  acquirer  for 
independent  code  review 

-  PMs  should  request  COTS  vendors  provide  Assurance 
Cases  for  their  COTS  products  detailing  both  the  vendor’s 
secure  coding  practices  and  the  results  of  internal  static 
code  analysis  or  third  party  assessment  (e.g.  Common 
Criteria  certification) 

-  In  cases  where  such  information  is  unavailable,  and  there  is 
still  a  desire  to  use  the  COTS  component,  the  PM  should 
consider  binary  code  analysis 

-  Such  analysis  could  be  performed  either  as  part  of  the 
system  integrator’s  life  cycle  process,  or  independently  by  an 
IV&V  agent. 

■  Ensure  that  a  party  other  than  the  developer  (such 
as  a  peer)  will  independently  perform  static  analysis 
and  test,  and  that  the  element  being  reviewed  will  be 
the  element  that  will  be  delivered. 
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Critical  Design  Review  (CDR) 

Objectives 


■  The  CDR  is  a  multi-disciplined  technical 
review  to  ensure  that  the  system  under 
review  can  proceed  into  system 
fabrication,  demonstration,  and  test, 
and  can  meet  the  stated  performance 
requirements  within  cost  (program 
budget),  schedule  (program  schedule), 
risk,  and  specific  assurance 
requirements  and  constraints. 
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Critical  Design  Review  (CDR) 

Outcomes 


■  Identification  and  use  of  selected  source  code  analysis  tools 

-  Selection  of  additional  development  tools  and  guidelines  to 
counter  weaknesses  and  vulnerabilities  in  the  system 
elements  and  development  environment(s) 

■  These  include  static  analysis  tools  for  source  code  evaluation. 

-  Definition  and  selection  of  assurance-specific  static  analyses 
and  assurance-specific  criteria  to  be  examined  during  peer 
reviews  performed  during  implementation. 

■  Documented  in  the  SEP  and  Software  Development  Plan  (SDP). 

-  Planning  for  training  for  assurance-unique  static  analysis 
tools  and  peer  reviews. 

-  Ensuring  that  another  party  (such  as  a  peer)  will 
independently  perform  static  analysis  and  test,  and  that  the 
element  being  reviewed  will  be  the  element  that  will  be 
delivered 

■  This  counteracts  the  risk  of  a  developer  intentionally  subverting 
analysis  and  test,  as  well  as  aiding  against  unintentional  errors. 

■  Assurance  Case 

-  Updating  of  the  assurance  case  with  relevant  evidence. 
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Test  Readiness  Review  (TRR) 

Objectives 


■  The  TRR  is  a  multi-disciplined 
technical  review  to  ensure  that  the 
subsystem  or  system  under  review  is 
ready  to  proceed  into  formal  test 
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Test  Readiness  Review  (TRR) 

Outcomes 


■  Verification  regarding  static  code  analysis 

-  Verification  that  assurance-specific  static  analysis  and  peer 
reviews  of  assurance  criteria  have  been  completed. 

-  Verification  that  another  party  (such  as  a  peer)  performed 
static  analysis  and  peer  review. 

-  Selection  of  any  additional  static  analysis  tools  to  identify  or 
verify  weaknesses  and  vulnerabilities  in  the  system  elements 
and  development  environment(s). 

-  Completion  and  verification  of  an  information  security 
technology  evaluation  for  all  critical  COTS/GOTS  elements. 

■  Open  source  verification 

-  Identification  of  industry  tools  and  test  cases  to  be  used  for 
the  testing  of  any  binary  or  machine-executable  open  source 
software  products  with  no  warranty  and  no  source  code. 

-  Documentation  of  evidence  that  static  analysis  has  been 
performed  (both  source  and  binary)  to  identify  weaknesses 
and  vulnerabilities  such  as  buffer  overruns  and  cross-site 
scripting  issues. 

■  Assurance  Case 

-  Updating  of  the  assurance  case  with  relevant  evidence 
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System  Verification  Review 
SVR/Production  Readiness  Review  (PRR) 

Objectives 


■  The  SVR  is  a  multi-disciplined  product  and  process 
assessment  to  ensure  that  the  system  under  review 
can  proceed  into  low-rate  initial  production  (LRIP) 
and  full-rate  production  (FRP)  within  cost  (program 
budget),  schedule  (program  schedule),  risk,  and 
other  system  constraints 

■  The  PRR  examines  a  program  to  determine  if  the 
design  is  ready  for  production  and  if  the  producer 
has  accomplished  adequate  production  planning 

■  The  primary  difference  between  PRR  and  TRR  is 
that  the  system  test  results  are  available  prior  to 
PRR 

-  If  changes  are  made  to  the  system  in  response  to  test 
results,  it  will  be  necessary  to  revisit  TRR  tasks 

-  Any  evidence  provided  by  system  test  results  should 
be  incorporated  into  the  assurance  case  prior  to  PRR 
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System  Verification  Review 
SVR/Production  Readiness  Review  (PRR) 

Outcomes 


■  Verification  regarding  static  code  analysis 

-  Verification  that  assurance-specific  static  analysis  and  peer 
reviews  of  assurance  criteria  have  been  completed. 

-  Verification  that  another  party  (such  as  a  peer)  performed 
static  analysis  and  peer  review. 

-  Selection  of  any  additional  static  analysis  tools  to  identify  or 
verify  weaknesses  and  vulnerabilities  in  the  system  elements 
and  development  environment(s). 

-  Completion  and  verification  of  an  information  security 
technology  evaluation  for  all  critical  COTS/GOTS  elements. 

■  Open  source  verification 

-  Identification  of  industry  tools  and  test  cases  to  be  used  for 
the  testing  of  any  binary  or  machine-executable  open  source 
software  products  with  no  warranty  and  no  source  code. 

-  Documentation  of  evidence  that  static  analysis  has  been 
performed  (both  source  and  binary)  to  identify  weaknesses 
and  vulnerabilities  such  as  buffer  overruns  and  cross-site 
scripting  issues. 

■  Assurance  Case 

-  Updating  of  the  assurance  case  with  relevant  evidence. 
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Operational  Test  Readiness 
Review  (OTRR)  Objectives 


■  The  OTRR  is  a  multi-disciplined  product 
and  process  assessment  to  ensure  that 
the  “production  configuration”  system 
can  proceed  into  Initial  Operational  Test 
and  Evaluation  with  a  high  probability  of 
successfully  completing  the  operational 
testing 

■  Successful  performance  during 
operational  test  generally  indicates  that 
the  system  is  suitable  and  effective  for 
service  introduction 
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Operational  Test  Readiness 

Review  (OTRR) 


■  Verification  regarding  static  code  analysis 

-  Re-verification  that  assurance-specific  static  analysis 
and  peer  reviews  of  assurance  criteria  have  been 
completed. 

■  Source  code  static  analysis  is  typically  not  performed  again 
for  OTRR,  but  binary  analysis  is  performed,  if  appropriate. 

-  Re-verification  that  another  party  (such  as  a  peer) 
performed  static  analysis  and  peer  review. 

-  Completion  and  verification  of  an  information  security 
technology  evaluation  for  all  critical  COTS/GOTS 
elements. 

■  Weaknesses  and  vulnerabilities  evaluation 

-  Documentation  of  evidence  that  the  system  has  been 
analyzed  for  weakness  and  vulnerabilities  using  static 
(binary)  analysis  tools  to  identify  such  flaws  as  buffer 
overruns  and  cross-site  scripting  issues 

■  Assurance  Case 

,  -  Updating  of  the  assurance  case  with  relevant  evidence 
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In-Service  Review  (ISR) 

Objectives 


■  The  ISR  is  a  multi-disciplined  product 
and  process  assessment  to  ensure  that 
the  system  under  review  is  operationally 
employed  with  well-understood  and 
managed  risk.  This  review  is  intended 
to  characterize  the  in-service  technical 
and  operational  health  of  the  deployed 
system.  It  provides  an  assessment  of 
risk,  readiness,  technical  status,  and 
trends  in  a  measurable  form 
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In-Service  Review  (ISR) 

Outcomes 


■  Configuration  Management 

-  Review  of  the  configuration  management 
process,  to  determine  that  it  remains  adequate 
with  respect  to  analysis  of  code  changes,  and 
being  followed 

■  Weaknesses  and  vulnerabilities  evaluation 

-  Documentation  of  evidence  that  any  changes  to 
the  software  throughout  its  service  life  have  been 
analyzed  for  weakness  and  vulnerabilities  using 
static  (source  or  binary)  analysis  tools  to  identify 
such  flaws  as  buffer  overruns  and  cross-site 
scripting  issues 

■  Assurance  Case 

-  Updating  of  the  assurance  case  with  relevant 
evidence 
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Challenges  to  Effective  Static 

Code  Analysis 
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Challenge  -  Procurement  and 
Maintenance  of  Tools 


■  The  better  static  code  analysis  tools  are 
expensive 

-  Use  multiple  tools  used  to  offset  tool  biases  and 
minimize  false  positives  and  false  negatives  can 
quickly  become  cost  prohibitive  for  a  single 
program 

-  In  addition,  maintenance  agreements  to  ensure  a 
tool  is  up  to  date  with  respect  to  the  spectrum  of 
threats,  weaknesses,  and  vulnerabilities  add  long 
term  costs 

■  Buy  it  once,  use  it  often  provides  the  most  bang 
for  the  buck 

■  Pooled-resources  analysis  labs  may  make 
economic  sense. 
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Challenge  -  Training 


■  Static  code  analysis  is  not  for  sissies,  although  it 
may  be  for  CISSPs  (Certified  Information  System 
Security  Professionals) 

-  This  tongue-in-cheek  statement  belies  the  difficulty  in 
using  static  code  analysis  tools  to  their  best  advantage 

-  Chandra,  Chess,  and  Steven  [12]  point  out  that  when 
static  code  analysis  tools  are  employed  by  a  trained 
team  of  code  analysts,  false  positives  are  less  of  a 
concern;  the  analysts  become  skilled  with  the  tools 
very  quickly;  and  greater  overall  audit  capacity 
results. 

■  In  order  to  determine  the  validity  of  static  code 
analysis  results,  it  is  important  for  PMs  to 
understand 

-  The  level  of  training  that  code  analysts  have  had  with 
the  tools  employed  for  static  code  analysis 

-  Their  understanding  of  code  weaknesses  and 
vulnerabilities 
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Useful  Links 


■  NIST  SAMATE  Static  Analysis  Tool  Survey 

-  The  National  Institutes  for  Science  and  Technology 
(NIST),  Software  Assurance  Metrics  and  Tool 
Evaluation  (SAMATE)  project,  provides  tables 
describing  current  static  code  analysis  tools  for  source, 
byte,  and  binary  code  analysis 

-  More  information  on  SAMATE  can  be  found  at 
http://samate.nist.gov/ 

■  DHS  Build  Security  In  Web  Site 

-  A  wealth  of  software  and  information  assurance 
information,  including  white  papers  on  static  code 
analysis  tools 

-  More  information  on  Build  Security  In  can  be  found  at 
https://buildsecuritvin.us-cert.gov/daisv/bsi/home.html 
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Commitment  to  Excellence  -  Enabling  acquisition  organizations  to  achieve  acquisition  excellence 
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□  Performance  Measurements 

>  Summary 


Objectives 


Support  Systems  Associates,  Inc. 


800  Park  Drive 


Warner  Robins,  GA  31088 


>  Illustrate  how  effective  Software  Engineering  Advisory 
and  Assistance  Services  enable  acquisition 
organizations  to  achieve  acquisition  excellence 

>  Provide  Key  Acquisition  Elements  for  enabling 
acquisition  excellence 

■  The  Contract,  The  Acquisition  Environment,  Requirements 
Management,  Risk  Management,  Technical  Performance 
Assessment,  Software  Test  Evaluation,  and  Performance 
Measurements 

>  Provide  detailed  Practical  Examples  from  major  military 
and  federal  programs 


Knowledge  of  failure  helps  lead  to  success 


A  Software  Acquisition  Journey 


Support  Systems  Associates,  Inc. 

_ Programs _ 

C-130  AMP 

Software  Engineering  Advisory 
and  Assistance  Services 

-  8  years 


800  Park  Drive  Warner  Robins,  GA  31088 

_ Roles _ 

Integrated  Product  Teams  Support 
Systems  Integration  Facility  (SIF) 
Operational  Flight  Program  (OFP)  Software 
Systems  Requirements,  Design  &  Test 


C-130J  Hercules 

Software  Subcontract 
Management 

-  4  years 


Supplier  Manager 
Review  and  approve  SDRL  items 
Monitor  supplier  activities 
Witness  acceptance  testing 
Coordinate  with  FAA  DER 


FAA  NAS  Plan  Programs 

Software  Engineering  Advisory 
and  Assistance  Services 

- 10  years 


System  Development  Manager  (AAS) 

SPO  Software  Lead  (TDWR) 

Software  Subject  Matter  Expert  (e.g.,  VSCS, 
MLS1,  RCE1,  NADIN  II,  MCCP/MCC2) 

1  Terminated  for  Default  Deposed  by  AT&T  (RCE),  GAO  Audit  (MLS) 

2  Terminated  for  Convenience 


Plus  a  foundation  of  19-years  Software  Development  and  Process  Improvement 

United  States  Patents  #4451702,  #4479034 


Examples  of  FAA  NAS  Programs 
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Advanced 
Automation  System 
(A  AS) 

Cornerstone  of  the 
NAS  Plan 

°  1984,  $276.7  million  Competitive  Design  Phase  Contract 
-  IBM  Federal  Systems  and  Hughes  Aircraft 
°  1988,  $3.6  billion  Fixed-Price,  -  IBM  Federal  Systems 
Statement  of  Work 

Q  Replace  computer  hardware  and  software  at  ATC 
facilities-Airport  Towers,  Terminal  Facilities,  and  En-Route 
Centers,  99.99999%  Reliability. 

Microwave  Landing 
System  (MLS) 

Q  1984,  $90.6  million  Fixed-Price  First  Production  - 
Hazeltine  Corporation 

System  Overview 

Q  Landing  aid  to  enable  planes  to  fly  a  wide  variety  of 
approach  paths  to  airport  runways. 

Radio  Control 
Equipment  ( RCE) 

°  1986,  Fixed-Price  Contract  (DTFA01-86-C-00034) 

-  AT&T  Company  Federal  Systems  Advanced  Technologies 
System  Overview 

°  Provides  pilots  communications  links  with  air  traffic 
controllers. 

Examples  of  FAA  NAS  Programs 
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Voice  Switching  and  Control 
System  (VSCS)  Upgrade 


q  1992-Contract  Award-$1 .3  billion,  Harris  Corporation 
System  Overview 

Q  Allows  air  traffic  controllers  to  communicate  with  pilots 
and  other  air  traffic  controllers  at  23  Air  Route  Traffic 
Control  Centers  (ARTCC) 

Q  Independent  distributed  processors  and  voice  switches, 
fault-tolerant  databases,  redundant  high-speed  bus 
interconnections,  operational  availability  -  0.9999999 


Terminal  Doppler  Weather 
Radar  (TDWR) 


°  1988,  Firm  Fixed-Price  Incentive  contract  -  Raytheon 
Systems  Company 

Develop,  produce,  and  install  47  TDWR  at  45  airport  sites 
System  Overview 

Q  Detects  and  reports  hazardous  weather  in  and  around 
airport  terminal  approach  and  departure  zones 
Q  Identifies  and  warns  air  traffic  controllers  of  low  altitude 
wind  shear  hazards  caused  by  micro-burst  and  gust  fronts 
Q  Reports  on  precipitation  intensities 
Q  Provides  early  warning  of  wind  shifts 
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Objectives 

A  Software  Acquisition  Journey 


oft  ware  Acquisition  Challerig 


Key  Acquisition  Elements 

□  The  Contract 

□  The  Acquisition  Environment 

□  Requirements  Management 

□  Risk  Management 

□  Technical  Performance  Assessments 

□  Software  Test  Evaluation 

□  Performance  Measurements 


Summary 


Software  Acquisition  Challenges 
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>  Why  is  Software  Acquisition  a  Challenge? 

□  Studies  have  shown  that  technical  performance,  cost ,  and 

schedule  risks  are  inherent  in  delivering  quality  software 
products  within  cost  and  schedule  constraints  AO  1 999] 

□  75%  of  all  large  scale  software  systems  fail 

■  [Software’s  Chronic  Crisis,  W  Wyat  Gibbs,  1994] 

□  Design  constraints  make  software  acquisition  and  development 
mission  critical 

-  Examples  of  design  constraints 

■  Application  domain  (real-time  embedded  systems  of  systems), 

■  Software  size 

■  Complexity,  Throughput/Timing 

■  High-integrity 

■  Reliability 

■  Safety-critical 

The  Software  Crisis  Is  Still  With  Us! 


Software  Acquisition  Challenges 
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>  Why  is  Software  Acquisition  a  Challenge? 

□  Software  size  is  the  critical  factor  in  determining  cost, 
schedule,  and  effort  [Jones  2004]  [Jones  1 999] 

■  Software  size  typically  driven  by  the  supplier’s  agreement 
terms  - 

■  Contract  vehicle  (Fixed-Price,  Cost-Reimbursement) 

■  Statement  of  work 

■  Deliverables  (Contract  Data  Requirements  List-CDRL) 

■  Technical  requirements  (safety-critical) 

■  Supplier’s  software  development  capability/maturity 

□  Software  Acquisition  Team  -  Inability  to  recognize 
quality  work 


“Acquirers  must  recognize  quality  work  before  they  can  require  and  accept  it” 


——Watts  Humphrey,  2009 


Examples  of  Acquisition  Problems 
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>  Cost  and  Schedule  Overruns 

>  Software  Performance  Issues 

□  Underestimate  software  size  and  complexity 

>  Lack  of  Software  Acquisition  Capability  Maturity 

□  Ability  to  specify  software  contractual  requirements 

■  Functional  and  Non-Functional 

□  Unable  to  recognize  product  quality 

□  Lack  of  software  expertise  in  acquisition,  project 
management,  and  the  application  domain 


I  Examples  of  Acquisition  Problems 

Support  Systems  Associates,  Inc. 
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FAA  NAS  Programs 

o  AAS 

o  Inadequate  requirement  baseline  control 
o  Cost  and  Schedule  Overruns 

o  Restructured  in  1994 

-  contract  cost  increased  from  $3.6 
billion  to  $7.6  billion 

o  NADIN  II 

o  Cost  and  Schedule  Overruns 

o  MCCP/MMC 

o  Termination  for  Convenience 

o  MLS 

o  Termination  for  Default 

o  RCE 

o  Termination  for  Default  (DOT  BCA 

No.  2479)  (FAR  52.249-8) 


Success  in  Acquisition 
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FAA  NAS  Programs 

o  TDWR1  0  Delivered  First  Production  Unit  sjx  months  ear/y 

o  Received  IEEE  Computer  Society  award 
o  Operational  at  45  Airports 

o  1991 ,  software  process  evaluated  a  SEI  CMM®  Level  3 

®  CMM  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University 


Acquirer  and  supplier  capability  /  maturity  levels  matched 


o  Production  completed 

o  100%  on-time  system  delivery  of  all  23  systems 

o  FAA  Contractor  of  the  Year  Award 
o  Human  Factors  Engineering  Society  Award 


o  VSCS 
Upgrade 


1  Successful  Acquisition  of  FAA  Terminal  Doppler  Weather  Radar ,  Third  Annual  Conference  on  the 
Acquisition  of  Software-Intensive  Systems  (Experience  Track,  26  January  2004).  [Jones  2004-1] 

http://www.sei.emu.edu/programs/acquisition-support/conf/2Q04-presentations/jones.pdf 
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□  The  Contract 

□  The  Acquisition  Environment 

□  Requirements  Management 

□  Risk  Management 

□  Technical  Performance  Assessments 

□  Software  Test  Evaluation 

□  Performance  Measurements 

>  Summary 


a  mams 


The  Contract 
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>  Contract  Administration 

>  Contract  Types 

□  Fixed-Price 

□  Cost-Reimbursable 

>  Contact  Data 

□  Statement  of  Work  (SOW)/Statement  of  Objective 
(SOO) 

□  Contract  Data  Requirements  List  (CDRL) 

□  System  Specification 

□  Data  Rights 


The  Contract  is  the  foundation  for  acquisition  success 


Contract  Administration 
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>  The  Contract  is  a  mutually  binding  legal  relationship  obligating  the 
seller  (supplier)  to  furnish  products  or  services  and  the  buyer 
(acquirer)  to  pay  for  them. 

>  Acquisition  management  involves  obtaining  products  or  services 
through  a  contractual  agreement. 

>  Contractual  authorif  -  delegated  to  an  Administrative  Contracting 
Officer  (ACO)/procuring  contracting  officer  (PCO) 


The  acquirer  specifies 
What  the  system  requires 
When  the  system  is  needed 
How  the  system  will  be 
accepted 


Concerns 


cost 
schedule 
technical 


The  supplier  determines 

•  How  the  system  will  be 
produced 

•  The  resources  required 
(examples) 

•  people,  equipment 

•  facilities 


The  degree  of  interaction  depends  on  the  nature  of  the  development  effort  and  the  type  of  contract 


Contract  Types 
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>  Basic  Compensation  Schemes  used  in 
Contracts 

□  Fixed-Price 

■  Acquirer  pays  the  supplier  a  fixed  sum 

■  The  supplier  assumes  the  risk 

■  Profit  is  a  direct  function  of  supplier’s  ability  to  deliver  the 
product  or  service 

□  Cost-Reimbursement 

■  Acquirer  agrees  to  reimburse  the  supplier’s  allowable  costs 
plus  profit 

■  The  risk  is  shared 


The  degree  of  acquirer/supplier  relationship  depends  upon  the  contract  type 


Contract  Data 
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>  Why  have  Contract  Data? 

□  Contract  vehicle  must  clearly  express  a  vision  of  the 
final  product  and  the  development  effort 

□  Software  acquisition  issues  must  be  addressed  in  the 
Request-For-Proposal  (RFP)  via  contract  data 

>  Key  Software-Related  Contract  Data  in  the  RFP 

□  Statement  of  Work  (SOW)/Statement  of  Objective 
(SOO) 

□  Contract  Data  Requirements  List  (CDRL)  Items 

□  System  Specification 

□  Data  Rights 


Success  of  an  acquisition  is  directly  linked  to  the  quality  of  the  RFP 

—  (Army  2007) 


sow/soo 
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>  What  is  the  Statement  Of  Work  (SOW)  /  Statement  Of 
Objectives  (SOO)? 

□  Basis  for  communicating  acquirer  requirements  to  the  supplier 

■  SOW  defines  specific  tasks 

■  SOO  defines  objectives 

□  Primary  document  for  translating  management  requirements  into 
contractual  tasks  /  objectives 

□  Sufficient  detail  must  be  provided  to  allow  the  supplier  to  scope 
the  effort,  cost  it,  and  provide  a  responsive  technical  solution 

□  Tasking  information  must  be  defined  for  the  preparation  of 
deliverable  data  (artifact) 

■  Each  tasking  statement  reference  applicable  Contract  Data 
Requirements  List  (CDRL)  item  which  will  be  delivered  by  that 
task. 


sow/soo 
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>  Examples  of  Key  Software  Tasking 

□  Software  development  process 

□  Software  management 

□  Software  engineering  —  software  requirements  analysis, 
Dreliminary  design,  detailed  design,  code  and  unit  test, 
ntegration,  and  formal  qualification  testing 

□  Software  tools  and  environment 

□  Risk  management 

□  Technical  reviews  —  Software  Specification  Review  (SSR), 
Preliminary  Design  Review  (PDR),  Critical  Design  Review  (CDR),  and 
Test  Readiness  Review  (TRR) 

□  Technical  Interchange  Meetings 

□  In  Process  Reviews 


The  SOW/SOO  tell  the  supplier  how  to  do  the  required  work 

The  SOW/SOO  specify  selection  of  major  software  components 


Contract  Data  Requirements  List 

CDRL 
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>  software  Data  (artifacts) 

□  Absolutely  essential  for  managing  the  development  process 

□  A  natural  by-product  of  the  development  effort  to  capture  results 
of  each  activity 

>  Contract  Data  Requirements  List  (CDRL)  Items 

□  Primary  vehicle  for  acquiring  software  data 

□  A  list  of  authorized  data  requirements  for  a  specific  procurement 
that  forms  a  part  of  the  contract. 

□  Defense  Federal  Acquisition  Regulation  Supplement  (DFARS) 
Subpart  215.470  Estimated  Data  Prices  requires  a  CDRL  (DD 
Form  1 423)  when  delivery  of  data  is  required 

□  CD  L  items  must  be  referenced  in  the  Statement  of  Work 
(SOW)  describing  the  development  effort 

□  Language  must  be  consistent  with  the  SOW 


Key  CDRL  Item  Requirements 
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Block 

Description 

4 

Authority  (Data  acquisition  Documentation  No.) 

Data  Item  Description  (DID1)  -  Defines  format  and  content  preparation 
instructions  for  data  product  generated  by  task  requirements 

Assist-Quick  Search  used  to  access  the  current  DID 

1  Should  be  tailored  to  meet  contract  requirements  (Block  16) 

5 

Contract  Reference  -  Reference  Statement  of  Work  paragraphs 

6 

Requiring  Office  -  Organization  have  primary  responsibility  for  reviewing 
the  data  and  recommending  acceptance/rejection  of  the  data 

8 

Approval  Code  -  (A)  Approved  by  the  Contracting  Officer 

Should  specify  approval  at  each  milestones  (e.g.,  SSR,  PDR,  CDR,  etc.) 

10,  11, 
12,  13 

Delivery  Requirements 

Should  be  associated  with  milestones  (e.g.,  SSR,  PDR,  CDR,  etc.) 

-  30  days  prior  to  the  milestone 

CDRL  Lessons  Learned 
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>  Software  CDRL  items  should  be  delivered  prior  to  the 
technical  reviews  to  allow  significant  time  to  enable: 

□  Acquirer  to  perform  a  detailed  review 

□  Supplier  to  disposition  the  review  comments 

□  Acquirer  to  provide  feedback  to  supplier  disposition 

>  echnical  review  should  include  review  of  supplier 
disposition  and  feedback 

>  software  CDRL  items  should  be  prepared  by  the 
software  team 

□  Reviewed  by  all  applicable  distribution  addressee  organization 

□  Approved  by  either  the  appropriate  Chief  Engineer,  Program 
Manager  or  Data  Requirements  Review  Board 


CDRL  Lessons  Learned 
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>  Typical  Software  CDRL  Items 

□  SOFTWARE  REQUIREMENTS  SPECIFICATION  (SRS)  -DI-IPSC-81433A 

■  Describes  the  behavior  of  the  software  to  be  developed  and  methods  to  be  used  to 
ensure  each  requirement  has  been  met 

■  Basis  for  the  design  and  qualification 

■  Interface  Requirements  Specification  (IRS)  -  DI-IPSC-81434A  may  be  appendix  to 
SRS 

□  SOFTWARE  DESIGN  DESCRIPTION  (SDD)  -  DI-IPSC-81435A 

■  Describes  the  design  and  detailed  design  needed  to  implement  the  software 

■  Interface  Design  Description  (IDD)-DI-IPSC-81436A,  may  be  appendix  to  SDD 

■  Database  Design  Description  (DBDD)-DI-IPSC-81437A,  may  be  appendix  to  SDD 

■  Describes  the  data  base  design  and  elements  (content  and  format) 

□  Software  Test  Plan  (STP)  -  DI-IPSC-81438A 

■  Describes  plans  for  qualification  testing,  test  environment,  identify  tests  to  be 
performed,  and  schedule 

□  Software  Test  Description  (STD)  -DI-IPSC-81439A 

■  Describes  the  test  preparation,  test  cases,  and  test  procedures  to  be  used  to  perform 
the  qualification  testing 

■  Enables  the  acquirer  to  access  the  adequacy  of  the  qualification  testing 

□  Software  Test  Results  (STR)  -  DI-IPSC-81440A 

■  A  record  of  the  qualification  testing 

■  Enables  the  acquirer  to  access  the  testing  and  its  results 

□  Software  Version  Description  (SVD)  -  DI-IPSC-81442A 

■  Identifies  and  describes  a  software  version  (“as-built”  software) 


System  Specification 
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>  What  is  the  System  Specification? 

□  Establish  top-level  technical  performance,  design, 
development,  integration,  and  verification 
requirements 

□  Examples  of  requirement  statements 

■  All  software  related  to  operation  in  civil  airspace  shall  be 
modified  or  developed  in  accordance  with  the  requirements 
of  RTCA  DO-178B  or  equivalent  level  of  safety 

■  All  newly  developed  software  shall  be  written  in  a  higher 
order  language  (HOL) 

■  Meteorological  algorithms  shall  be  implemented  in  high  order 
language  (HOL) 

•  Use  of  commercial  software  shall  be  approved  by  the  FAA 


Data  Rights 
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>  Data  Rights 

□  Enable  the  use,  maintenance,  and  replication  of  the  software  data 

>  Data  Rights  Categories 

.  Unlimited  rights  -  right  to  use,  modify,  reproduce,  release,  in  whole  or 
in  part,  in  any  manner  and  for  any  purpose  whatsoever,  and  to  have  or 
authorize  others  to  do  so.  Software  developed  exclusively  with  acquirer 
funds. 

□  Acquirer  Purpose  rights  -  rights  to  use,  modify,  reproduce,  release, 
within  the  acquirer’s  organization/company  without  restriction.  Software 
development  with  mixed  acquirer  and  supplier  funding. 

□  Restricted  data  rights  apply  only  to  noncommercial  computer  software 
and  mean  that  the  acquirer’s  rights  are  as  set  forth  in  a  Restricted 

Rights  Notice.  Supplier  funds  all  development . _ 

Secretary  of  the  Air  Force  Memo  -  Data  Rights  and  Acquisition  Strategy  (3  May  06)  - 


directing  the  acquisition  of  technical  data  and  associated  rights  to  be  addressed 
specifically  in  all  Acquisition  Strategy  Plans,  reviews,  and  associated  planning 
documents  for  Acquisition  Categories  (AC AT)  programs  -  software  intensive  systems 
and  subsequent  source  selections. 


Content 
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>  Objectives 

>  A  Software  Acquisition  Journey 

>  Software  Acquisition  Challenges 

>  Key  Acquisition  Elements 


□  The  Contract 


□  Requirements  Management 

□  Risk  Management 

□  Technical  Performance  Assessments 

□  Software  Test  Evaluation 

□  Performance  Measurements 

>  Summary 


The  Acquisition  Environment 
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Typical  Software 

Products 

•  Software  Development  Plan 
(SDP) 

•  Software  Configuration 
Management  Plan  (SCMP) 

•  Software  Quality  Assurance 
Program  Plan  (SQPP) 

•  Software  Requirements 
Specification  (SRS) 

•  Interface  Requirements 
Specification  (IRS) 

•  Software  Design  Description 
(SDD) 

•  Interface  Design  Description 
(IDD) 

•  Software  Test  Plan  (STP) 

•  Software  Test  Description 
(STD) 

•  Software  Test  Report  (STR) 

•  Software  Version  Description 
(SVD) 


Best  Practices:  Better  Matching  of  Needs  and  Resources,  will  lead  to  Better  Weapon  Systems  Outcomes. ..GAO  2001 


Acquirer 
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>  Acquirer  Capability  Maturity 

□  Software  acquisition  team  must  have  software  expertise  in 
application  domain,  acquisition,  process,  project  management, 
engineering,  and  safety,  as  needed 

□  A  software  lead  must  be  designated  to  be  responsible  for 
establishment  and  managing  the  software  acquisition  activities 

□  The  software  acquisition  team  must  have  adequate  resources 
and  funding  to  perform  the  acquisition  activities 

□  The  software  acquisition  team  must  be  trained  (Examples) 

■  Software  Acquisition  Management 

■  Application  domain  (Radar,  Communications  Systems,  etc) 

■  Processes,  Procedures,  Standards  being  used 

■  Technologies,  Tools,  Methodology  being  used 


“Acquirers  must  recognize  quality  work  before  they  can  require  and  accept  it ” 

——Watts  Humphrey 


Example  of  FAA  AAS  SPO 
Svstem  Development 
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Supplier 
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>  Supplier  Capability  Maturity 

□  A  set  of  software  process  assets  must  be  established  and 
maintained 

□  The  project  must  develop  a  defined  software  process  by  tailoring 
the  organization’s  standard  processes 

□  Software  plans  (software  development  plan  (SDP),  software 
configuration  management  plan,  and  software  quality 
assurance  plan)  must  be  documented  and  institutionalized 

□  The  SDP  must  provide  the  acquirer  with: 

■  Insight  into  the  processes,  procedures,  and  desk  instructions 

■  Tools  and  methods  used 

□  Development  environment  must  be  augmented  by  management 
practices 

■  Measuring  and  monitoring  progress 

■  Judging  the  quality  of  the  software 

■  Validating  the  deliverable 

■  Conducting  technical  reviews  and  in-process  reviews 


Typical  Software  Process  Definition 
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>  Objectives 

>  A  Software  Acquisition  Journey 

>  Software  Acquisition  Challenges 

>  Key  Acquisition  Elements 


□  The  Contract 

□  The  Acquisition  Environment 


□  Risk  Management 

□  Technical  Performance  Assessments 

□  Software  Test  Evaluation 

□  Performance  Measurements 


>  Summary 


Requirements  Management 
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>  Requirements  change  for  variety  of  reasons 

□  Additional  requirements  are  derived  or  changes  made  to  the 
existing  requirements 

>  Requirements  Management  involves  establishing  and 
maintaining  bidirectional  traceability  of  requirements, 
design,  source  code,  and  test  to  ensure  the  right  product 
is  being  built 

>  Bidirectional  traceability  is  required  by  CDRL  item  DID 

>  Bidirectional  traceability  is  essential  for  Safety  Critical 

>  Supplier  must  manage  changes  and  identify  any 
inconsistencies 

>  Supplier  must  track  measures  of  requirements  volatility 


Requirements  management  is  fundamental  to  a  controlled  and  disciplined 
engineering  design  process  [CMMI 2006] 


Bidirectional  Traceability 
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>  Required  by  the  CDRL 
item  DID 

>  Allocation  ensures  the 
right  products  been  built 

>  Reduce  effort  required  to 
determine  change  impact 

>  Traceability  ensures  the 
evolving  product  is  not 
expanding  the  scope 

>  Should  be  Documented  in 
a  requirements  database 

□  DOORS®,  RTM 


Bidirectional  traceability 


©DOORS  is  a  trademark  of  Telelogic  AB 


Content 


Support  Systems  Associates,  Inc.  800  Park  Drive  Warner  Robins,  GA  31088 


>  Objectives 

>  A  Software  Acquisition  Journey 

>  Software  Acquisition  Challenges 

>  Key  Acquisition  Elements 


□  The  Contract 

□  The  Acquisition  Environment 

□  Requirements  Management 


□  Technical  Performance  Assessments 

□  Software  Test  Evaluation 

□  Performance  Measurements 


>  Summary 


Risk  Management 
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>  Why  Manage  Risks? 

□  ;isk  is  like  fire:  if  controlled  it  will  help  you;  if  uncontrolled  it  will  rise  up 
and  destroy  you... 

■  Theodore  Roosevelt 

□  Technical  performance,  cost,  and  schedule  risks  are  inherent  in 
software  intensive  systems  development  [GAO  1999] 

□  One  key  obstacle  is  the  inability  to  see  cost  and  schedule  issues  as 
symptoms  of  unforeseen  problems 

■  Software  size  growth,  requirements  growth,  complexity,  ability  to  perform 

□  Air  Force  expects  the  acquisition  communities  to  address  Risk 
Management  throughout  the  life  cycle  of  the  acquisition  program  [DoD 
2004] 

■  Continuously  identify  and  manage  risks 

■  Ensure  the  risks,  impact,  and  mitigation  plans  are  appropriately  addressed 
during  program  reviews. 

□  Risk  Management  is  a  process  element  of  the  1 0  Life  cycle  Processes 
of  Operational  Safety  Suitability  and  Effectiveness  [AFMC  63-1201] 

■  1)  Risk  Management  Planning,  2)  Risk  Identification,  3)  Risk  Assessment,  4) 
Identification  of  Risk  Options,  5)  Decision  Analysis,  6)  Implementation,  and 
7)  Risk  Monitoring 


Risk  Management 


Support  Systems  Associates,  Inc.  800  Park  Drive 

>  Managing  Risks 

□  Establish  a  Risk  Management  Model  to 
define  a  systematic  process 

□  Establish  consistent  Risk  Statement  to 
allow  recognition  of  the  impact  or 
consequence 

□  Establish  a  Risk  Information  System  for 
identifying,  analyzing,  planning,  tracking, 
and  controlling  risk. 

□  Risk  Information  System  should  include  - 
storage  media,  the  procedures,  and  the 
tools  for  accessing  the  risk  system 


Warner  Robins,  GA  31088 


Example  of  Risk  Management 
Model  ---[Van  Scoy  1992], 

Tools 

•  MITRE 

•Risk  Matrix 
•Risk  Management 
Toolkit 

•  AFMC  TAMC  20071 

•Probability  of  Program 
Success  (PoPS) 
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>  Objectives 

>  A  Software  Acquisition  Journey 

>  Software  Acquisition  Challenges 

>  Key  Acquisition  Elements 

□  The  Contract 

□  The  Acquisition  Environment 

□  Requirements  Management 

□  Risk  Management 


□  Software  Test  Evaluation 

□  Performance  Measurements 


>  Summary 


Technical  Performance  Assessment 
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>  How  to  Reduce  the  Risks,  Increase  the  Reliability  and  Quality,  and 
Ensure  Compliance  with  Requirements 

□  Software  work  products  (artifacts)  are  absolutely  essential  for 
managing  the  development  process 

□  Gaining  adequate  visibility  into  the  supplies’  process,  plans,  and 
software  products  is  key  to  technical  performance  assessments 


>  Technical  Performance  Assessment  provide: 

□  Visibility  into  the  process,  quality  and  reliability  of  the  software 
products. 

□  Feedback  to  improve  the  software  process 

□  Ensures  compliance  with  requirements 

□  Key  technical  performance  assessments 

■  Process 


Progress 
Software  Product 


Acquirers  must  recognize  quality  work  before  they  can  require  and 

accept  it 

— Watts 

Humphrey,  2009 
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>  Process  Assessment  -  Ensure  software 
management,  engineering,  configuration 
management,  and  quality  assurance  activities 
compliance  with  contractual  requirements  and 
supplier’s  defined  software  process  and  plans 

Process  Assessment  key  focus  is  “what  is  done  and  the  product  being  built” 

>  Examples  of  Software  Plans 

□  Software  Development  Plan  (SDP) 

□  Software  Configuration  Management  Plan  (SCMP) 

□  Software  Quality  Assurance  Plan  (SQAP) 


The  Contract  must  provide  mechanism  to  gain  access  to  process  and  plans 


Technical  Performance  Assessment 


Support  Systems  Associates,  Inc. 


800  Park  Drive  Warner  Robins,  GA  31088 


>  Progress  Assessment  conducted  to  determine  what  is  done 

□  Contract  SOW  must  specify  Technical  Reviews  and  Design  Reviews  to 

be  held  to  determine  progress,  status,  surface  issues,  and  provide 
feedback.  Examples: 

■  Technical  Reviews  (Examples) 

■  Program  Management  Review 

■  Program  Configuration  Control  Boards 

■  Technical  Interchange  Meeting 

■  In-Process 

■  Design  Reviews  -  used  as  quality  gates  (progress  and  quality) 

■  (e.g.,  Software  Specification  Review  (SSR),  Preliminary  Design  Review 
(PDR),  Critical  Design  Review  (CDR),  etc) 

□  Supplier  must  conduct  informal  reviews  such  as  Peer  Reviews  in 
accordance  with  supplier’s  defined  process 

□  Acquirer  must  participate  in  Technical  Reviews  and  Design  Reviews  to 

■  Gain  visibility  into  the  progress  and  status 

■  Discuss  issues/candidate  risks 

■  Provide  feedback 


Technical  Performance  Assessment 
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>  Software  Products  Assessment 

□  Supplier  must  evaluate  CDRL  items  prior  to  delivery 
and  place  under  configuration  control 

□  Supplier  should  deliver  CDRL  items  prior  to  the 
technical  review  to  allow  significant  time  for  detailed 
review  and  disposition  of  review  comments 

■  CDRL  delivery  and  review  comments  disposition  must  be  the 
entrance  criteria  for  the  technical  review 

□  Acquirer  must  establish  a  CDRL  review  process 

□  Acquirer  must  complete  the  review  within  an  agreed 
upon  time  after  receipt  of  the  CDRL  items 


Software  Product  Assessment 
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>  Acquirer  typical  review  process 

□  Evaluation  CDRL  using  evaluation  criteria 

□  Evaluation  criteria  examples 

■  Compliance  with  DID  format  and  content 

■  Completeness  (e.g.,  missing  requirements,  testing,  interfaces,  etc.) 

■  Traceability  (e.g.,  test  traced  to  requirements,  etc.) 

■  Consistency  with  upper  level  documents 

■  Internal  consistency 

■  Ambiguity  of  requirements  (understandable,  testable?) 

■  Conflicting  requirements 

■  Test  coverage  of  requirements 

■  Appropriate  analysis,  design,  and  coding  techniques  used 

□  Provide  discrepancies  and  recommendations  to  supplier 

□  Conduct  meeting  with  supplier  to  disposition  supplier  responses. 


Practical  Examples 
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>  FAA  NAS  (TDWR)  Contract 

□  16  CDRL  Items  specified  by  the  SOW 

□  Submittal  (preliminary  and  final)  linked  to  design 
review  (e.g.,  SSR,  PDR,  etc) 

□  Acquirer  approval  within  30-calendar  days 

>  Raytheon 

□  45  Total  CDRL  Items  delivered 

>TDWR  Software  IPT 

□  Over  4300  Review  Items  Discrepancies  approved 
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□  Technical  Performance  Assessments 


□  Performance  Measurements 
>  Summary 
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>  What  is  Software  Testing? 

□  Software  development  involves  a  series  of  activities  in 
which  opportunities  for  human  induced  defects  are 
enormous 

■  46%  -  60%  of  all  software  defects  originate  in  the  software 
requirements  analysis  phase  [Endves  1975]  [Voges  1979] 

□  Software  Testing  is  the  quality  assurance  technique 
used  to  evaluate  the  “as-built”  software  product  to 

ensure  the  probability  of  failure  due  to  latent  defects 
is  low  enough  for  acceptance 

□  Software  testing  typically  consists  of  three  levels  of 
testing 

■  Unit  Testing ,  Integration,  and  Formal  Qualification  Testing 


Software  testing  represents  the  ultimate  evaluation  of  the  software  requirements,  design,  and 
coding  activities  [Jones  1993-1] 

Software  testing  can  make  the  software  product  more  reliable  and  usable  [Musa  1987]  [Dunnl984] 
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>  What  is  required  in  the  Contract? 

□  Unit  Testing,  Integration,  and  Formal  Qualification  Testing  (FQT) 
activities  and  artifacts  must  be  documented  in  the  supplier’s 
defined  software  process  and  the  Software  Development  Plan 

□  FQT  activities  and  artifacts  must  be  specified  in  the  SOW 
Examples 

■  Planning  -  Software  Test  Plan  (CDRL  item) 

■  Test  Description  -  Software  Test  Description  (CDRL  item) 

■  Test  Cases  and  Test  Procedures 

■  Test  Results  -  Software  Test  Report  (CDRL  item) 

□  Test  Readiness  Review  (TRR)  must  be  held  prior  to  FQT 
execution  to  determine  readiness 

□  Software  test  artifact  must  be  delivered  at  designated  quality 
gates  (i.e.,  PDR,  CDR,  TRR,  and  Product  Release) 

>  Acquirer  and  Supplier’s  Software  Quality  Assurance 
must  witness  all  FQT  execution 


Software  Test  Evaluation 
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>  Problem  Reporting/Tracking 

□  Supplier  process  must  be  institutionalized  to: 

■  Document  problems  identified  during  FQT  and  to  track  the 
problems  to  ensure  closure 

■  Determine  the  severity  of  all  problems  detected 

■  Control  changes  to  the  software  products  under  configuration 
control 

■  Analyze  the  changes  to  determine  impact  to  the  work 
product,  related  work  product,  and  schedule 

■  Analyze  the  problem  closure  to  determine  the  impact  to  the 
software  release  milestone 


Change  control  system  should  be  used  to  determine  the  aspects  of  process 
impro  vement  and  effectiveness  of  previous  activities 


Typical  Problem  Report  Life  Cycle 
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>  How  much  testing  is  enough? 

□  Complete  test  coverage  is  generally  not  possible 
[Jones  1993-1] 

□  Test  Case  design  methodology  must  be  documented 

□  Acquirer  and  supplier  must  mutually  agree  on 
completion  criteria  Examples 

■  Completion  of  a  number  of  test  runs  with  no  open  priority  1 
and  2  severity  problems 

□  Acquirer  and  supplier  should  establish  a  failure 
intensive  objective  (FIO)  using  a  software  reliability 
growth  model:  Examples 

■  Time-Between-Failure  Models 

■  Error-Count  Model 


Acquirer  and  supplier  face  a  difficult  decision  when  to  release  the  software  product 
Complete  test  coverage  is  generally  not  possible. .  .[Jones  1993-1] 
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>  Why  Measure  Performance? 

□  Software  development  is  often  out-of-control.  You 
cannot  control  what  you  cannot  measure. .  .[DeMarco 
1982] 

□  Performance  Measurement  is  key  to  managing  and 
producing  quality  software  and  is  an  essential 
element  of  software  process  improvement  [Humphrey 
1989] 

□  National  Defense  Acquisition  Act  Section  804-2003 
mandate 

■  Metrics  for  performance  measurement  and  continual  process 
improvement 


Performance  Measurement 
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>  How  to  Measure  Performance? 

□  Software  Measures  should  be  captured  to  document 
actual-versus-plan  and  to  identify  problems 

□  Software  Measures  should  be  selected  that  are 
directly  measurable  to  evaluate  progress  and  identify 
significant  predictors  [Jones  2004] 

□  Software  Measures  should  be  selected  to  provide 
insight  into  four  key  acquisition  areas: 

■  Process  -  insight  into  the  software  development  process 
and  how  it  is  working 

■  Product  -  insight  into  the  quality  of  the  product  (frequency  of 
requirement  changes,  number  of  problems,  review 
comments) 

■  Project-  schedule  attainment,  CDRL  delivery 

■  Productivity  -  rate  at  which  the  work  is  progressing 


Performance  Measurement 
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>  How  to  use  Software  Measures? 

□  Provide  overview  of  development  progress 

□  Early-warning  for  detecting  process  and  quality  issues 

□  Provide  feedback  to  refine  the  process  and  contribute 
to  positive  control 

>  Typical  software  measures 

■  Software  size 

■  Cost/Schedule  deviation 

■  Schedule  progress 

■  Activity  progress 

■  Requirements  stability 

■  Resource  tilization 

■  Documentation  (Artifact)  review  item  discrepancies 


Examples  of  Performance  Measures 
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Cost/Schedule  Deviation 
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FQT  Progress 


Development  Progress 


MLS  Software  Detailed  Design 


Mymbw  gf  CSUs  Completed 


1906 


NSS-  Node  Software  Specification 


Simulation  Hardware:  Node  A  &  SIL 
Simulation  Software:  EXEC  CSCI 

LRU  CSCI  NIS-  Node  Interface  Specification 
ENV  CSCI  NTP-  Node  Test  Plan 

NTD-  Node  Test  Description 
SRS-  Software  Requirements 
SDET-'Sblfware  Design  Description 


NES(AOII)  NSS  (A01 2)  NIS(A013)  NTP(A016)  NTD  (A017)  SRS  (A012)  SDD(A014)  STP(A016)  NHS  (N/A) 


Document  Review  Item 
Discrepancies 


TDWR  Software  Development  Documentation 


RID& 


1.238 


SOP  STP  SRS  IRS  5DD  IDO  STB  STR 
C DEL  Items 


CDRL  Items 
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Achieving  acquisition  excellence... 

>  Receiving  quality  software  delivered  on  time 

□  THE  CONTRACT  must  specify  what  is  required 

□  THE  ACQUISITION  TEAM  must  have  the  acquisition 
capability  maturity  to  perform 

■  “Acquirers  must  recognize  quality  work  before  they  can 
require  and  accept  it”  -—Watts  Humphrey,  2009 

■  The  acquirer  can  negatively  impact  the  supplier 

□  RISK  MANAGEMENT  must  be  performed  to  control 
the  inherent  performance,  cost,  and  schedule  risks 

□  PERFORMANCE  MEASUREMENTS  must  be 
performed  to  control  the  development  activities 


Summary 
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>  Reducing  the  risks,  increasing  the  reliability,  and 
quality 

□  TECHNICAL  PERFORMANCE  ASSESSMENTS 
must  be  performed  to  gain  insight  into  the  process 
and  product  quality 

-  Identify  discrepancies  in  the  process  and  products 

-  Provide  feedback  to  disposition  of  discrepancies 

-  Vehicle  for  process  improvement 

□  SOFTWARE  TEST  EVALUATION  must  be 
performed  to  ensure  the  “as-built”  software  product 
meets  software  requirements 

□  REQUIREMENT  MANAGEMENT  must  be  performed 
to  ensure  the  right  product  is  being  built  at  each 
phase  throughout  the  lifecycle 
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>  Improvements  in  Software  Acquisition 

□  Public  Law  107-314  Section  804  of  the  National 
Defense  Authorization  Act,  released  in  December 
2002  [Section  804-2003] 

□  Clinger-Cohen  Act:  Initiatives  such  as  Software 
Assurance  and  Open  Architecture 


□  The  best  practice  model  Capability  Maturity  Model® 
Integration  (CMMI®)  for  Acquisition 


■  rhttp://www.whitehouse.qov/the  press  office/Memorandum-for-the-Heads-of- 

Executive-Departments-and-Aqencies-Subiect-Government-Contractinq/  1 


®  Capability  Maturity  Model,  CMM,  CMM  Integration,  and  CMMI 
Registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University 
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Support  Systems  Associates,  Inc. 
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Email:  ijonesffissai.ora 
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Process,  1999  Joint  ISPA/SCEA  Conference,  1999 

>  Conforming  to  ISO  9001:  A  Mission  Success  Solution  to  Product  Development, 

Lockheed  Martin  Management  and  Data  Systems,  1997 

>  Software  Metrics  Effectiveness  in  Software  Acquisition  Management,  38th  Air  Traffic 
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AAS 

Advanced  Automated  System 

FQT 

Formal  Qualification  Testing 

ACAT 

Acquisition  Category 

IDD 

Interface  Design  Description 

AMP 

Avionics  Modernization  Program 

IRS 

Interface  Requirements  Specification 

ATC 

Air  Traffic  Control 

MP 

Mission  Processor 

CDR 

Critical  Design  Review 

NAS 

National  Airspace  System 

CDRL 

Contract  Data  Requirements  List 

OFP 

Operational  Flight  Program 

CIP 

Capital  Investment  Plan 

OFP 

Operational  Flight  Program 

CNS/ATM 

Communications/Navigation  Surveillance  /  Air  Traffic 

PCO 

Procuring  Contracting  Officer 

Management 

PDR 

Preliminary  Design  Review 

CO 

Contracting  Officer 

SCM 

Software  Configuration  Management 

COTS 

Commercial  Off-The-Shelf 

SDD 

Software  Design  Description 

CPAF 

Cost-Plus  Award  Fee 

SOF 

Special  Operations  Forces 

CSCI 

Computer  Software  Configuration  Item 

SOO 

Statement  of  Objective 

CY 

Calendar  Year 

SOW 

Statement  of  Work 

DCI 

Document  Comment  Item 

SPO 

System  Program  Office 

DER 

Designated  Engineering  Representative 

SQA 

Software  Quality  Assurance 

DFARS 

Defense  Federal  Acquisition  Regulation  Supplement 

SRS 

Software  Requirements  Specification 

DID 

Data  Item  Description 

SSR 

Software  Specification  Review 

DoD 

Department  of  Defense 

STD 

Software  Test  Description 

DOORS 

Dynamic  Object-Oriented  Requirements  Systems 

STP 

Software  Test  Plan 

ECP 

Engineering  Change  Proposal 

STR 

Software  Test  Report 

EMD 

Engineering,  Manufacturing  and  Development 

SVD 

Software  Version  Description 

FAA 

Federal  Aviation  Administration 

TRR 

Test  Readiness  Review 

FFP 

Firm  Fixed-Price 

FFPI 

Firm  Fixed-Price  Incentive 

Toward  Best  Practices  in 
Contracting  for  M&S 

Presentation  to  the  12th  Annual 
Systems  Engineering  Conference 


CNA 

ANALYSIS  &  SOLUTIONS 


COTS 

GOTS 

Dennis  Shea 
Julie  Nelson 

29  October,  2009 


Work  in  progress 


Business  Models 


Plans  for  Acquisition  and  Requirements  Reform 


“DoD  will  expand  and  improve  training  programs  in  critical  risk  areas 
such  as  program  execution,  source  selection,  risk  management, 
pricing  and  contracting  ”  DoD  will  also  increase  resident  training, 
expand  simulations,  and  continue  to  leverage  e-learning  technologies 
to  improve  its  ability  to  deliver  learning  assets  at  the  employee’s  point 
of  need.” 


Memo  for  Assistant  to  the  President  for  National  Security  Affairs 

Robert  M.  Gates  (SecDef) 
April  6,  2009 
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How  we  got  here 


M&S  Reuse  Business  Model  recommendation 

-  To  strengthen  reuse  DoD  should: 

•  Improve  M&S-related  contracting  practices 

Understand  implications  of  procurement  statues,  regulations, 
policies  etc  on  contracting  for  M&S-related  products 

-  Protect  government  “rights”  to  reuse,  modify,  share,  M&S 
resources) 

-  Develop  RFPs  with  reuse  in  mind 

Monitor  M&S  development  process  (avoid  proprietary 
markings) 

•  Incorporate  best  practices  in  training  for  contract 
officers  and  PMs  and  other  Gov’t  M&S  decision-makers 

Match  intended  uses  of  M&S  products  and  services  with 
contracted  deliverables  including  data  rights 
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Objective 


•  Improve  DoD  M&S  by  publishing  best  practices  for  M&S- 
related  contracting  from  a  government  perspective 

-  Address  full  scope  of  contracting  activities 

-  Procurement  of 

-  M&S  products  (including  databases)  and  services  (including 
configuration  management)  and  support  tools 

Larger  project  (weapon  system,  training  service)  where  M&S  is  but 
an  element 

-  Provide  a  roadmap  to  existing  best  practices  and  fill  in 
gaps 

-  Use  front  line  experiences  of  acquisition  force  to  assess  what’s 
working  well  and  what’s  not 

-  Use  analysis  to  recommend  changes  to  existing  regulations, 
policies,  and  guidance  to  overcome  challenges  and  promote 
efficiencies  in  contracting  (benefit  both  government  and  industry) 


CNA 
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Background:  Identified  8  activities  in  acquiring  M&S 


Determine  M&S  capabilities  required  to  successfully  execute  the 
acquisition,  planning,  training,  testing  etc.  task(s) 

Identify  well-defined  M&S  capability  attributes  to  ensure  that  capability 
delivered  meets  user  needs  at  an  affordable  cost 

Use  Discovery  to  determine  if  existing  M&S  resources  satisfy 
required  capabilities  and  are  available  for  reuse 

3.  Source  Selection  Planning 

-  Identify  gaps  in  needed  M&S  capabilities 

Determine  how  best  to  satisfy  requirements  (new  development, 
existing  commercial,  or  improved  GOTS) 

-  Assess  potential  for  reuse  and  role,  if  any,  for  proprietary  products 

Develop  source  selection  criteria  (including  maturity  of  firm’s  M&S 
capabilities) 

Decide  on  level  of  data  rights  required  and  compliance  with  DoD 
policy  on  open  architectures,  HLA,  VV&A,  etc 
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Activities  in  acquiring  M&S  (continued  3-5) 


Source  Selection  Planning  (continued) 

Assess  interoperability  and  data  interchange  requirements  and 
standards  needed  to  satisfy  requirements 

Perform  market  survey  to  gauge  qualifications  of  commercial  vendors 
and  government  labs  to  develop  needed  M&S 

4.  Develop  solicitation  for  M&S  capabilities 

-  Use  results  from  source  selection  planning  to  develop  RFP 
Proposal  Evaluation  and  Contract  Award 

-  Assess  “best  value”  for  government 
Small  business  concerns 

-  Type  of  contract 

Special  contract  terms  and  conditions 
Negotiate  license  terms  and  data  rights 
Negotiate  deliverables 
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Activities  in  acquiring  M&S  (continued  6-8) 


6.  Contract  administration  post  award 

-  Logbook  of  vendor’s  alternative  funding  sources  for  software 
employed  to  determine  data  rights 

-  Resolving  M&S  to  lowest  component  level  (data  rights) 

-  Markings  on  software  and  database  products 

-  Government’s  responsibilities  in  protecting  proprietary  data 

7.  Acceptance  of  contract  deliverables 

-  Source  Code,  User  Guide,  Analyst  Guide,  VV&A 

-  Quality  and  timeliness  to  support  current  (future?)  users 

8.  Contracting  for  life-cycle  maintenance  of  the  M&S 
resource 

-  Including  multi-user  licenses  and  other  technical  data  rights 

-  Including  configuration  management 

-  Central  funding  for  broadly  used  tools  vs  Program  or  PEO- 
unique  needs 
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Today’s  discussion:  Selecting  M&S  with  reuse  in  mind 
{Activities  3-5) 


•  General  context:  DoD  IT  acquisitions,  particularly  M&S 
software  (in  context  of  a  business  model) 

•  Policy:  Use  commercial  products  whenever  appropriate 

and 

Assess  the  long-term  technical  data  needs 

•  Issues  to  resolve: 

-  Factors  determining  default  rights  ownership 

-  Ambiguous  guidance  on  appropriate  scope  of  rights 

-  Contracting  context:  licensing  v.  ownership  of  data  rights 

•  Contract  language  examples 

-  Problematic  language 

-  Consistent  language 
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DoD  FY08  IT  obligations  for  products  &  services 


(millions) 

percent 

GSA  Product  or  Service  Code 

$10,938 

57% 

D  --  IT  services,  including  telecommunications  services 

$6,305 

33% 

70  --  General  purpose  information  technology  equipment 

$783 

4% 

J  --  Maintenance,  repair  &  rebuilding  of  equipment 

$379 

2% 

R  -  Professional,  admin,  and  mgmt  support  services 

$378 

2% 

Other  IT  items 

$260 

1% 

Other  IT  services 

$19,043 

100% 

Total 
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Policy:  Emphasize  “commercial  software” 


United  States  Code:  10  U.S.C.  2377(a) 

•  The  head  of  an  agency  shall  ensure  that  procurement  officials  in 
that  agency,  to  the  maximum  extent  practicable,  acquire 
commercial  items  or  nondevelopmental  items  other  than 
commercial  items  to  meet  the  needs  of  the  agency 

Appropriations  Legislation:  FY09  NDAA 

•  The  Secretary  of  Defense  shall  ensure  that  contracting  officials 
identify  and  evaluate,  at  all  stages  of  the  acquisition  process 
(including  concept  refinement,  concept  decision,  and  technology 
development),  opportunities  for  the  use  of  commercial 
computer  software  and  other  non-developmental  software. 
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Policy  on  “commercial  software”  (continued) 


DoD  Directive  5000.01  -  “Most  preferred”  option  for 
acquisition: 

•  El  .1 .1 8.1 .  The  procurement  or  modification  of  commercially 
available  products,  services,  and  technologies,  from  domestic 
or  international  sources,  or  the  development  of  dual-use 
technologies. 

DoD  Instruction  5000.02 

•  6.  DoD  ENTERPRISE  SOFTWARE  INITIATIVE.  When  the  use 
of  commercial  IT  is  considered  viable,  maximum  use  of  and 
coordination  with  the  DoD  Enterprise  Software  Initiative  shall  be 
made. 
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Importance  of  “data  rights”  in  IT  acquisitions 


GAO  (2006)  finding: 

•  “The  lack  of  technical  data  rights  has  limited  the  services’ 
flexibility  to  make  changes  to  sustainment  plans  that  are  aimed  at 
achieving  cost  savings  and  meeting  legislative  requirements 
regarding  depot  maintenance  capabilities.” 

Legislative  Response  -  FY07  NDAA: 

•  The  Secretary  of  Defense  shall  require  program  managers  for 
major  weapon  systems  and  subsystems  of  major  weapon 
systems  to  assess  the  long-term  technical  data  needs  of  such 
systems  and  subsystems  and  establish  corresponding  acquisition 
strategies  that  provide  for  technical  data  rights  needed  to  sustain 
such  systems  and  subsystems  over  their  life  cycle. 
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Factors  determining  default  data  rights  ownership 


•  Relevant  set  of  regulations:  FAR  v.  DFARS 

•  Nature  of  product  or  service  to  be  acquired:  commercial 
v.  non-commercial 

•  Nature  of  contracting  process:  commercial  v.  non¬ 
commercial 

•  Nature  of  contracting  vehicle  (contract,  IDIQ,  BPA,  FSS, 
grant,  SBIR,  OTA,  “special  work”,  etc) 

•  Specific  terms  of  agreement 
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IT  contract  funding  obligations  -  FY08 


(millions) 

Item  or  service 

$4,883 

Commercially  available  item 

$932 

Other  commercial  item 

$961 

Non-developmental  item 

$789 

Non-Commercial  item 

$7,575 

Commercial  service 

$3,904 

Non-commercial  service 

$19,043 

Grand  total 

Commercial 

procedures 

Used 

Commercial 
procedures  not 
used 

77% 

23% 

50% 

50% 

1% 

99% 

5% 

95% 

58% 

42% 

3% 

97% 

46% 

54% 

Grand 

total 

1 00% 

1 00% 

1 00% 

1 00% 

1 00% 

1 00% 

1 00% 
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DFARS  guidance  on  acquiring  data  rights 


Some  regulations  limit  the  scope  of  rights  to  be  acquired: 

DFARS  227.71 02-1  (a):  Commercial  technical  data 

-  DoD  shall  acquire  only  the  technical  data  customarily  provided  to  the 
public  with  a  commercial  item  or  process 

DFARS  227.71 03-1  (a):  Noncommercial  technical  data 

-  DoD  policy  is  to  acquire  only  the  technical  data,  and  the  rights  in  that 
data,  necessary  to  satisfy  agency  needs. 

DFARS  227.7202  and  227.7203  specify  similar  terms  for  software 
acquisitions 
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USD(AT&L)  guidance  on  “Other  Transactions” 


Other  guidance  specifies  a  maximal  acquisition  of  data  rights 
under  certain  type  of  agreements 

•  C2.3.3.3.2.  Allocation  of  Rights.  The  agreement  must 
explicitly  address  the  government’s  rights  to  use,  modify, 
reproduce,  release,  and  disclose  the  relevant  technical  data 
and  computer  software.  The  government  should  receive  rights 
in  all  technical  data  and  computer  software  that  is 
developed  under  the  agreement,  regardless  of  whether  it  is 
delivered,  and  should  receive  rights  in  all  delivered  technical 
data  and  computer  software,  regardless  of  whether  it  was 
developed  under  the  agreement. 
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Current  Practices 


•  What  works? 

•  What  can  be  improved? 

•  Sources  of  information: 

-  Recent  RFPs  -  downloaded  from  INPUT  (FedBizOps) 

-  Recent  contracts 

-  Stakeholder  interviews 

-  Regulations,  guidance  and  research 


CNA 
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Sample  INPUT  RFP  header 


LIVE  VIRTUAL  CONSTRUCTIVE  INTEGRATED  ARCHITECTURE  AND  INFRASTRUCTURE 
PROGRAM  (LVC  IA) 

Not  Marked 


Opportunity  ID  :  S1733  Mark  as: 
Opportunity  Summary 


sfsfsfsfsf© 


CNA 


Department: 

ARMY 

Agency: 

ASST  SEC  FOR  ACQUISITION,  LOGISTICS  AND 
TECHNOLOGY 

Office: 

PEO  SIMULATION „  TRAINING  AND  INSTRUMENTATION 

Status: 

Pre-RFP 

Solicitation  Release 

Date: 

10/2009  [INPUT  Estimate) 

Award  Date: 

04/2010  [INPUT  Estimate) 

Solicitation  #: 

PEOSTRI  KOCLVd  A 

Value{SK): 

65,100 

Competition  Type: 

GW  AC  /  MAC  /  MAIDIQ 

Primary  Requirement: 

System  Design  and  Architecture 

Duration: 

2  yeaa(s)  base  plus  4x1  yea r[s)  option(s) 

Contract  Type: 

Primary  NAICS  Code: 

Indefinite  Delivery  Indefinite  Quantity 

Place  of  Performance: 

Q Hand dr  FL 

Opportunity  Website: 

Click  to  Visit  Website 

Updated  : 

09/09/2009 

Procurement  Timeline 

Registration 

12/01/2008 

Conference 

12/03/2008 

Related  Details 


Key  Contacts 

Milton  Washington  ESI 

Contracting  Officer 

PEO  SIMULATION, 
TRAINING  AND 
INSTRUMENTATION 

407-38 1 -83  54 

Donald  Stewart  ESI 

Administrative  Point  of 
Contact 

PEO  SIMULATION, 
TRAINING  AND 
INSTRUMENTATION 

407-384-5333 

Dave  Bukovey  22 

Prog  ram  Manager 

PEO  SIMULATION, 
TRAINING  AND 

INSTRU  MENTATION 

407-208-3398 

More  >* 

10  Total 
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RFP  and  contract  examples 


•  TRADOC  Support:  (Contract  No:  GS-00F-0014N) 

•  Modeling  And  Simulation  and  Analytically  Based  Warfare  Analyses 
(Solicitation  N00024-09-R-3145) 

•  Marine  Corps  Studies  System  (Solicitation  M00264-06-D-0006) 

•  NMCI  Transition  Plan 

•  System  Development  and  Demonstration  (SDD)  Phase  for  the 
Broad  Area  Maritime  Surveillance  (BAMS)  Unmanned  Aircraft 
System  (UAS)  Program  (Solicitation  00019-07-R-0001) 

•  Engineering  Technical  And  Support  Services  For  Naval  Warfare 
Center  And  Other  NAVSEA  Field  Sites  (Solicitation  N00178-04-R- 
4000) 

•  F-16  Mission  Training  Center  (MTC)  Program  (Solicitation  FA8621- 
07-R-6291) 

•  BLCSE  TECHNICAL  SUPPORT  EFFORT  -  Solicitation  W91 2SU- 
07-R-0002 

CNA 
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Insights 


CNA 


Problematic  language 


Very  broad  data  rights  claims  by  the  government: 

Contract  Example^  1 7.0.  Proprietary  Rights.  All  proprietary  rights  to 
all  materials  produced  by  Contractor  personnel  shall  become  the 
sole  property  of  the  US  Government. 

RFP  Example :  As  such,  no  analysis  or  data  provided  under  this 
contract  shall  constitute  or  be  construed  as  company  proprietary  or 
owned  by  the  contractor.  Upon  termination  of  the  contract,  all 
related  hardware,  software,  data  and  materials  shall  become 
property  of  the  United  States  Government  and  shall  not  be  disclosed 
except  upon  written  authority  by  the  Task  Order  Manager  (TOM). 


CNA 
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Problematic  language:  inconsistent  clauses 


Contract  example: 

H-13  Software  and  data  rights 

H-13.1.  The  Government  shall  be  provided  with  unlimited  rights  to  all 
data  and  automated  models  developed  in  support  of  this  contract  in 
accordance  with  the  policy  expressed  in  DFARS  227.71 ,  and  the 
requirements  of  DFARS  Clause  252.227-7015,  Technical  Data  - 
Commercial  Items,  and  DFARS  Clause  252.227-  7013,  Rights  to 
Technical  Data  -  Noncommercial  Items.  Proprietary  models  must  not 
be  used  for  any  task  order  under  this  contract  without  the  specific, 
written  approval  by  the  COR,  prior  to  start  of  any  work. 

Problems : 

•  The  commercial  items  clause  (‘701 5)  cited  above  does  not  specify 
unlimited  data  rights  for  the  government. 

•  All  of  the  clauses  cited  explicitly  exclude  computer  software 
(contrary  to  the  implication  of  the  section  title). 

CNA 
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Problematic  language  (continued) 


Contract  example: 

H-14.2.  DFAR22.227-7013  (June  1995)  Rights  in  technical  data  and 
computer  software  shall  apply  to  those  monthly  reports  and  other 
documents  that  are  not  characterized  as  "Special  Works." 

Problems : 

•  Clause  DFAR22. 227-701 3  does  not  exist. 

•  Clause  DFARS  252.227-701 3  does  not  apply  to  computer  software. 


CNA 
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Problematic  language:  Transition  planning 


1.  DEFINITIONS. 

a.  Contractor  Proprietary  Software  means  computer  software  that  is  owned 
by  Contractor  and  used  solely  by  Contractor  in  providing  ZZZZ  Services, 
and  includes  Related  Software  Documentation... 

f.  Licensed  Software  and  Other  Intellectual  Property  means  Contractor 
Proprietary  Software  and  Other  Intellectual  Property  for  which  the 
Government  has  negotiated  and  paid  for  a  license  to  use. 

k.  Other  Intellectual  Property  means  intellectual  property,  other  than 

proprietary  software,  relating  to  the  design  and  operation  of  the  ZZZZ  that  is 
owned  by  Contractor. 

2.4  MARKING  OF  CONTRACTOR  INTELLECTUAL  PROPERTY  AND 
PROPRIETARY  INFORMATION. 

Contractor  shall  conspicuously  and  legibly  mark,  and  the  Government 
shall  not  remove,  the  below  legend  on  hard  or  soft  copies  of  documents 
and  other  tangible  embodiments  of  Contractor  Proprietary  Software  and 
Other  Intellectual  Property  to  which  such  access  or  license  is  granted  to 
Government. 


CNA 
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Problematic  language  ^transition  planning  (continued) 


3.6  TRANSITION  OF  CONTRACTOR  PROPRIETARY  SOFTWARE  AND 
OTHER  INTELLECTUAL  PROPERTY. 

(a)  Government  Option  to  License  Contractor  Proprietary  Software  and 
Other  Intellectual  Property:  Government  or  its  Successor  Contractor(s) 
may  desire  rights  to  Use  the  Contractor  Proprietary  Software  and  Other 
Intellectual  Property  to  perform  the  ZZZZ  Services  after  this  Contract 
expires  or  is  terminated.  The  Government  may  elect  to  acquire  from 
Contractor  a  specifically  negotiated  license  to  use  all,  or  a  part  of, 
Contractor’s  Proprietary  Software  and  Other  Intellectual  Property  upon 
the  terms  and  conditions  set  forth  herein. 


(d)  License;  Reservation  of  Rights.  If  Government  elects  to  a  acquire  license 
to  use  the  Licensed  Software  and  Other  Intellectual  Property,  the 
Government  and  Contractor  shall  enter  into  a  mutually  acceptable  license 
agreement  pursuant  to  which,  upon  payment  of  the  mutually  agreed  upon 
license  fee  (the  “License  Fee”),  said  license  shall  become  effective. 
Contractor  reserves  all  rights  not  expressly  granted  in  the  license. 
Nothing  in  this  Contract  conveys,  or  shall  be  construed  to  convey,  any 
right  of  ownership  of  Contractor  Proprietary  Software  and  Other 
Intellectual  Property  to  Government  or  to  its  Successor  Contractor(s). 
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Consistent  language  -  scope  of  rights 


RFP  Example :  Technical  Data  Package/Rights  - 

•  Provide  the  appropriate  SOW  tasking  in  Annex  F  for  delivery  of  the 
technical  data  and  rights  sufficient  to  accomplish  the  objectives  in  SOO 

paragraph  3.1 .3.4  and,  in  this  paragraph,  demonstrate  the  extent  to  which  the 
objectives  can  be  met. 

•  Provide  a  description  of  the  level  of  data  and  rights  separately  with  regard  to 
(a)  meeting  Title  10  U.S.C.,  Chapter  146,  Section  2464,  CORE  logistics 
capabilities  (CLIN  0101);  (b)  a  competitively  selected  PBL  support  environment 
(CLIN  0102);  and  (c)  the  source  data  and  data  rights  for  a  mission  control 
system  to  enable  use  and  modification  for  other  UASs  (CLIN  0103). 

•  For  each,  the  Offeror  shall  propose  their  approach  for  providing  technical 
data  from  both  the  prime  and  subcontractors,  with  sufficient  rights  and  licenses 
provided  to  the  Government.  Specifically  identify  the  data  and  the  rights  that  will 
be  provided,  clearly  identifying  the  additional  data  and  rights  to  be  provided  that 
are  above  what  will  be  provided  as  part  of  the  effort  under  CLINs  0002  and  0202. 
Also,  if  any,  identify  data  and  rights  that  are  required  to  meet  the  objectives  but 
will  not  be  provided. 
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Consistent  language  -  classifying  deliverables 


RFP  Example:  Intellectual  Property  Deliverable  Restrictions. 

For  each  task  order  to  be  issued  under  the  contract,  the  Contractor 
shall  identify,  prior  to  award  of  the  affected  task  order(s)  to  the  best 
of  its  ability,  noncommercial  and  commercial  technical  data  and 
computer  software  that  it  intends  to  deliver  with  restrictions  on  the 
Government's  right  to  use,  release  or  disclose  such  identified 
technical  data  and/or  computer  software. 


CNA 
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Consistent  language  -  Cost  of  data  rights 


RFP  Example:  7. 2.2.9  Software  Parametric  Data. 

...Software  costs  shall  be  included  in  the  Basis  of  Estimate  (RFP 
Section  J  Attachment  18),  including,  license  costs  for  COTS  items, 
and  costs  for  providing  Government  unlimited  rights  for  software 
developed  by  the  offeror  and  its  suppliers. 
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Lessons  learned  thus  far: 


•  Request  appropriate  level  of  data  rights  (don’t  ask  for  the 
universe!) 

•  Use  contract  language  that  recognizes  the  contractor’s 
copyright  ownership 

•  Discuss  data  rights  issues  at  the  beginning  of  the 
contract 

•  Plan  for  a  smooth  transition  at  the  end  of  a  contract 
(avoid  a  hostage  crisis) 

•  Consider  a  repository  (virtual)  to  hold  data  rights 
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Ongoing  activities 


•  Extend  analysis  of  RFPs  and  contracts  to  requirements, 
contractor  qualifications,  deliverables,  ... 

•  Interview  PMs,  M&S  leads,  Gov  model  mgrs,  COR 

•  Survey  industry  for  recommended  best  practices 

•  Suggest  clarifying  and  simplifying  language  for 
acquisition  guidance 
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CNA 

ANALYSIS  &  SOLUTIONS 


Backup  slides 


What  distinguishes  a  “Best  Practice”  for  M&S  Contracting? 


—  • 


•  Complies  with  established  policy,  regulations,  statutes 

•  Avoids  post-award  protests 

•  Achieves  PM’s  objectives 

-  Program  cost/schedule/performance 

•  M&S  completes  on  time/schedule 

•  Leads  to  successful  use  of  M&S 

•  Balances  government  and  industry  interests 

•  Supports  broader  DoD  M&S  aims 

-  Reuse,  agile,  adaptable,  open,  interoperable 
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Contracting  context:  works  of  federal  employees 


The  U.  S.  Code  does  not  extend  protection  to  works  of  the 
federal  government: 

•  17  USC  §  105:  Copyright  protection  under  this  section  is 
not  available  for  any  work  of  the  United  States 
Government... 

-  A  “work  of  the  United  States  Government”  is  a  work 
prepared  by  an  officer  or  employee  of  the  United  States 
Government  as  part  of  that  person's  official  duties. 

Thus  we  must  look  elsewhere  to  identify  the  data  rights 
retained  by  federal  contractors. 


CNA 


33 


Contracting  context  works  of  federal  contractors 

17  USC  §  105:  ...but  the  United  States  Government  is  not  precluded 
from  receiving  and  holding  copyrights  transferred  to  it  by  assignment, 
bequest,  or  otherwise. 

In  other  words,  the  ownership  of  technical  data  rests  with  the 
contractor  unless  explicitly  transferred. 

However,  DFARS  does  not  necessarily  require  this  assignment: 

227.7103-4  License  rights  for  non-commercial  technical  data. 

•  (a)  Grant  of  license.  The  Government  obtains  rights  in  technical  data, 
including  a  copyright  license,  under  an  irrevocable  license  granted  or 
obtained  for  the  Government  by  the  contractor.  The  contractor  or 
licensor  retains  all  rights  in  the  data  not  granted  to  the 
Government.  For  technical  data  that  pertain  to  items,  components,  or 
processes,  the  scope  of  the  license  is  generally  determined  by  the 
source  of  funds  used  to  develop  the  item,  component,  or  process. 
When  the  technical  data  do  not  pertain  to  items,  components,  or 
processes,  the  scope  of  the  license  is  determined  by  the  source  of 
funds  used  to  create  the  data. 
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Problematic  language 


Overly  broad  data  rights  claims  by  the  government: 

Contract  Example^  1 7.0.  Proprietary  Rights.  All  proprietary  rights  to 
all  materials  produced  by  Contractor  personnel  shall  become  the 
sole  property  of  the  US  Government. 

RFP  Example :  As  such,  no  analysis  or  data  provided  under  this 
contract  shall  constitute  or  be  construed  as  company  proprietary  or 
owned  by  the  contractor.  Upon  termination  of  the  contract,  all 
related  hardware,  software,  data  and  materials  shall  become 
property  of  the  United  States  Government  and  shall  not  be  disclosed 
except  upon  written  authority  by  the  Task  Order  Manager  (TOM). 


CNA 


35 


Consistent  language  -  classifying  deliverables 


RFP  Example:  Intellectual  Property  Deliverable  Restrictions. 

For  each  task  order  to  be  issued  under  the  contract,  the  Contractor 
shall  identify,  prior  to  award  of  the  affected  task  order(s)  to  the  best 
of  its  ability,  noncommercial  and  commercial  technical  data  and 
computer  software  that  it  intends  to  deliver  with  restrictions  on  the 
Government's  right  to  use,  release  or  disclose  such  identified 
technical  data  and/or  computer  software. 
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GOTS  vs  COTS  Business  Models 


Positive 

GOTS 

•  Designed  to  meet  specific 
requirements 

•  Information  Assurance  (IA)  built  in 

•  Fewer  barriers  to  reuse  (Unlimited 
rights) 

COTS 

•  Potential  for  reduced 
cost/schedule 

•  Potential  for  multiple  alternatives 

•  Leverage  improvements  from 
broader  marketplace 

•  Leverage  experiences  fm  wider 
user  base 

CNA 


Negative 

•  Potential  cost/schedule  overruns 

•  Potential  loss  of  innovation 


•  May  not  fully  meet  program 
requirements 

•  Greater  integration  risks  with 
multiple  vendors 

•  No  visibility  inside  product 

•  Vendor  stability 

•  Dependence  on  license 
agreements 

•  DoD  unable  to  dictate  product 
improvements 


DFARS  license  rights  for  technical  data  &  software 


•  DFARS  227.7103-9(a)(1):  The  clause  at  252.227-7013,  Rights  in 
Technical  Data-Noncommercial  Items,  requires  a  contractor  to 
grant  or  obtain  for  the  Government  license  rights  which  permit  the 
Government  to  reproduce  data,  distribute  copies  of  the  data,  publicly 
perform  or  display  the  data  or,  through  the  right  to  modify  data, 
prepare  derivative  works. 

•  DFARS  227.7203-9(a)(1):  The  clause  at  252.227-7014,  Rights  in 
Noncommercial  Computer  Software  and  Noncommercial  Computer 
Software  Documentation,  requires  a  contractor  to  grant,  or  obtain  for 
the  Government  license  rights  which  permit  the  Government  to 
reproduce  the  software  or  documentation,  distribute  copies,  perform 
or  display  the  software  or  documentation  and,  through  the  right  to 
modify  data,  prepare  derivative  works. 
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Data  rights  under  FAR 


27.403  Data  rights — General. 

•  All  contracts  that  require  data  to  be  produced,  furnished,  acquired, 
or  used  in  meeting  contract  performance  requirements,  must  contain 
terms  that  delineate  the  respective  rights  and  obligations  of  the 
Government  and  the  contractor  regarding  the  use,  reproduction,  and 
disclosure  of  that  data. 

27.404-3  Copyrighted  works 

•  (a)(1 )  Generally,  the  contractor  must  obtain  permission  of  the 
contracting  officer  prior  to  asserting  rights  in  any  copyrighted  work 
containing  data  first  produced  in  the  performance  of  a  contract. 
However,  contractors  are  normally  authorized,  without  prior  approval 
of  the  contracting  officer,  to  assert  copyright  in  technical  or  scientific 
articles  based  on  or  containing  such  data  that  is  published  in 
academic,  technical  or  professional  journals,  symposia  proceedings 
and  similar  works 
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Data  rights  under  FAR  52.227-14 


(2)  The  Contractor  shall  have  the  right  to — 

•  ji)  Assert  copyright  in  data  first  produced  in  the  performance  of  this  contract  to  the  extent  provided 
in  paragraph  (c)(1 )  of  this  clause; 

•  (ii)  Use,  release  to  others,  reproduce,  distribute,  or  publish  any  data  first  produced  or  specifically 
used  by  the  Contractor  in  the  performance  of  this  contract,  unless  provided  otherwise  in 
paragraph  (d)  of  this  clause; 

(c) (1)  Copyright —  (1)  Data  first  produced  in  the  performance  of  this  contract. 

•  (i)  Unless  provided  otherwise  in  paragraph  (d)  of  this  clause,  the  Contractor  may,  without  prior 
approval  of  the  Contracting  Officer,  assert  copyright  in  scientific  and  technical  articles  based  on  or 
containing  data  first  produced  in  the  performance  of  this  contract  and  published  in  academic, 
technical  or  professional  journals,  symposia  proceedings,  or  similar  works.  The  prior,  express 
written  permission  of  the  Contracting  Officer  is  required  to  assert  copyright  in  all  other  data  first 
produced  in  the  performance  of  this 

(d)  Release,  publication,  and  use  of  data. 

The  Contractor  shall  have  the  right  to  use,  release  to  others,  reproduce,  distribute,  or  publish  any 
data  first  produced  or  specifically  used  by  the  Contractor  in  the  performance  of  this  contract, 
except — 

•  (1 )  As  prohibited  by  Federal  law  or  regulation  (e.g.,  export  control  or  national  security  laws  or 
regulations); 

•  (2)  As  expressly  set  forth  in  this  contract;  or 

•  (3)  If  the  Contractor  receives  or  is  given  access  to  data  necessary  for  the  performance  of  this 
contract  that  contain  restrictive  markings,  the  Contractor  shall  treat  the  data  in  accordance  with 
such  markings  unless  specifically  authorized  otherwise  in  writing  by  the  Contracting 
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Data  rights  for  project  involving  OneSAF 


•  3.3.1  Within  scope  of  TICC  and  BEST  activities  (paragraph  3.1 .1 
above)  the  contractor  shall  modify,  integrate  and/or  install 
simulation-independent  HLA  Middleware  software  packages, 
Gateways,  Adaptors  or  Interfaces,  permitting  existing  BLCSE 
simulations  to  interact  with  3CE  and/or  Lead  System  Integrator  (LSI) 
federates.  Middleware,  Gateways,  Adaptors  and  Interfaces  not 
provided  as  GFE  will  be  developed  and  delivered  with  complete 
documentation,  operating  instructions,  and  with  complete 
government  proprietary  rights.  OOS-based  or  developed,  or  other 
GFE  available,  Middleware,  Gateways,  Adaptors  and  Interfaces  will 
be  utilized  unless  prior  written  agreement  is  received  from  COTR 
that  no  OOS  or  other  GFE  Middleware,  Gateways,  Adaptors  and 
Interfaces  are  available  or  suitable  to  the  integration  task. 


CNA 


41 


Data  rights  for  project  involving  NSS,  STORM 


RFP  Example:  As  such,  no  analysis  or  data  provided  under  this 
contract  shall  constitute  or  be  construed  as  company  proprietary  or 
owned  by  the  contractor.  Upon  termination  of  the  contract,  all 
related  hardware,  software,  data  and  materials  shall  become 
property  of  the  United  States  Government  and  shall  not  be  disclosed 
except  upon  written  authority  by  the  Task  Order  Manager  (TOM). 


CNA 
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Process  management  and  tool 
selection  to  minimize  risk  of 
hand-arm  vibration  syndrome 


NDIA  Systems  Engineering  Conference- 
October  26-29,  San  Diego,  CA 

Mark  Geiger,  MS.,  CIH,  CSP 

Navy  Safety  Liaison  Office  (OPNAV  N09FB)/Naval  Safety  Center 

Presented  by 
Mr.  Sherman  Forbes 

SAF/AQRE  Acquisition  ESOH  Risk  Management 
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Outline 


Hand-arm  vibration  (HAV)  -  Background 

-  Under-recognized  occupational  disease 

-  Potential  for  prevention 

Defense  Safety  Oversight  Council 

-  Project  objectives 

Anti-vibration  gloves 
Power  tools 


Challenges 


What  is  Hand-Arm  Vibration? 


Energy  into  the  hands/arms  from  vibrating  tools 
Important  Factors: 

-  magnitude 

-  direction 

-  frequency 


What  is  the  Deal? 


Hand/arm  vibration  exposure  can  be  excessive  in  the 
workplace 

Many  highly  exposed  groups  have  incidence  of  disease  in 
the  range  of  1 0  to  50% 

Poorly  recognized  -  improvements  often  limited  or  absent 

-  Quarry  workers  studied  in  1918  has  80%  incidence  of  disease 

-  Follow-up  in  late  1970s  showed  same  tool,  similar  disease 
incidence  and  included  some  grandson’s  of  original  study  group 

Many  of  the  exposures  can  be  reduced  significantly. 
Lowering  hand/arm  vibration  can  have  several  benefits 


Health  Effects 
Hand-Arm  Vibration  (HAV) 

Syndrome 

Disease  States: 

Reynaud's  Phenomenon  of 
Occupational  Origin 

Carpal  Tunnel  Syndrome 

Bone  and  Joint  Disorders 

Neurological  Disorders 


Ilands  of  vibrating  pneumatic 
hand-tool  operator  in  later  stages  of 
irreversible  Hand  Arm  Vibration 

Syndrome  1 


Hand  Power  Tool  Use  in  the  Department  of  Defense 


TOOL  TYPE 

PRIMARY  PROCESSES  INVOLVED 

Maritime  / 
Shipyard 

Construction 

Aircraft  and 
Vehicles 

Mx 

Ground/Road  and 
Facility 
Maintenance 

Forestry 

Mining/ 

Milling/ 

Quarry 

Grinders 

X 

X 

X 

Polishing 

Limited 

Limited 

XX 

Welding  and  Pre-Post 
Grinding 

XX 

X 

X 

Limited  to 
Support  Ops 

X 

Mechanical  Metal  Cutting 

X  Submarine 
Recycling 

XX 

X 

X  Concrete  Work 

XX 

Wood  Cutting/Finishing 

X  (support 
structures) 

XX 

X 

XX  Chain 
Saws 

X  (Support 
Structures) 

Concrete  Work;  Finishing  and 
Set-up,  Cutting 

XX 

Mixers, 

Jackhammers 

Impact  Wrenches 

X 

X  Riveting  and 
Airframes 
Maintenance 

XX  Tires 
and  Wheel 

X 

XX  Assembly 

Demolition 

X 

XX  Jackhammers 

X  (Tree 
Stump  and 
Rock 
Removal) 

XX 

Foundry  Operations  and 
“Finishing"  Cast  Work 

X 

Limited 

Support 

areas 

Drilling 

X 

XX 

XX 

XX 

X 

XX 

Stone  Cutting 

XX 

XX 

X 

XX  Quarry 
WoriT 

Metrics  and  Outcome 


Metrics  &  Outcome:  The  occupational  exposure 
limits  for  hand-arm  vibration  demonstrate  a  very 
good  correlation  between  exposures  to  vibration 
(measured  as  acceleration)  and  the  incidence/ 
Drevention  of  disease.  An  example  from  the 
:orestry  industry  is  provided  below  (Koskimies  et 
al  1992) 


Equipment  type  (Chain  Saw)  Vibration 


Existing  equipment  (unimproved)  14  m/s2 
(1972) 

Anti-vibration  design  2  m/s2 


Prevalence 
of  HAV 

40% 

5%  (1990) 
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Hand-Arm  Vibration  Standards 


•  ISO  5349-1986 

Guidelines  for  measurement  and  evaluation 

•  ISO  8662-5-1992 

Handle  measurement  pavement  breakers/hammers 

•  ANSI  S3. 34-1 986 

Guidelines  for  measurement  and  evaluation 

•  ACGIH-TLV 

Guidelines  for  evaluation  and  control 
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ACGIH 


Total  Daily  Exposur 
Duration 

4  hours  and  less  than  8 


2  hours  and  less  than  4 


1  hour  and  less  than  2 


Less  than  1  hour 


Vibration  TLV 


e 

Acceleration  Level 
(m/s2) 

4 

6 

8 

12 

10 


Discussion 


Productivity  increases  when  vibration/ 
ergonomic  equipment/tools  are 
incorporated  into  a  process 
Injuries  and  disability  are  expensive, 
quality  of  life  diminished 
Side-benefit:  better  quality  products 


Defense  Safety  Oversight  Council  (DSOC) 
Hand-Arm  Vibration  Project  Task  Objectives 

•  Provide  procurement  guidelines  for  anti-vibration 
gloves  and  power  hand  tools  that  will  reduce 
personnel  exposure  to  crippling  hand-arm 
vibration  exposures  while  reducing  noise 
exposures  and  promoting  process  efficiency 
(Completed  Feb  08) 

•  Support  GSA/DLA  procurement  of  special  anti¬ 
vibration  gloves  which  reduce  the  vibration 
transmitted  to  the  fingers  and  hands  during  tool 
use  (In  process,  required  information  provided) 


12 


Defense  Safety  Oversight  Council  (DSOC) 
Hand-Arm  Vibration  Project  Task  Objectives 

•  Support  the  Federal  (GSA/DLA)  procurement  of 
more  modern  designs  for  powered  hand  tools 
meeting  current  performance  criteria  for  reduction 
of  transmitted  vibration  to  the  hands  when  in  use 
(Ongoing) 

•  Incorporate  criteria  for  3rd  party  evaluation  of 
vibration  for  gloves  and  tools  into  procurement 
criteria  (Completed  Feb  08) 

•  Communicate  this  information  to  logistics  and 
safety  communities  via  DLA,  GSA,  NIOSH  and 
Service  websites  (Linked  to  updated  product 
availability) 


DSOC  Project  Team 


Army 

Navy 

Headquarters  U.S.  Coast  Guard 
Air  Force  Research  Lab 
Defense  Logistics  Agency,  Headquarters 
Government  Services  Administration 
Contract  Support 

-  Coordinated  by  Concurrent  Technologies  Corporation  for 
OSD  Personnel  and  Readiness  (P&R) 

-  Don  Wasserman  (Vibration  expert) 

-  Robbins  Gioia  (Logistics  Contractor) 


Anti-Vibration  Gloves  (AVG): 

The  Problem 

•  Many  gloves  marketed  as  AVG  do  not  meet  the 
criteria  of  ISO  10819/ANSI  S2.73 

-  These  include  2  products  in  the  GSA  system  as 
National  Stock  Number  (NSN)  items 

•  There  are  no  US  regulations  for  manufacturers 
to  test,  certify,  and  label  gloves  that  meet  the 
ISO/ANSI  criteria 

•  Products  currently  marketed  by  GSA  as  “anti¬ 
vibration  gloves”  do  not  meet  these  criteria 
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AVG:  The  Approach 

Develop  procurement  criteria  consistent  with  anti¬ 
vibration  standard  and  incorporate  into  GSA 
procurement  (Completed  at  NIOSH  meeting  2-08) 

-  Evaluate  compliance  with  ANSI  S2.73  for  all 
gloves  intended  for  use  where  vibration  is  a 
hazard 

-  Develop  estimates  of  glove  use  from  current 
glove  National  Stock  Numbers  (completed  5-08) 

Develop  a  plan  to  address  the  need  for  AVG  and 
ways  to  procure  only  ANSI  S2.73  compliant  gloves 


Getting  Certified  Anti-Vibration  Gloves  in 

Supply  System 

Two-year  effort  requiring 

-  Intervention  of  DLA  Headquarters,  OSD  Manpower  and  Personnel 

-  Support  of  Navy  Clothing  and  Textile  Research  Facility,  Natick,  MA 

-  Defense  Logistics  Information  Service  cataloging 
Process  challenges  included 

-  Poorly  described  process 

-  Differences  in  motivation  among  supply  contacts 

-  Challenges  in  “new”  vendors  gaining  access  to  established  supply 
channels 

-  Buy-American  requirements-  overcome  by  vendors  willing  to 
produce  American-made  products  at  slightly  higher  costs 

Certified  Anti-Vibration  Gloves  (photos  and  sources  of), 

http://safetvcenter.navv.mil/acauisition/vibration/downloads/Anti- 

Vibration  Gloves.pdf 
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Power  Tools:  The  Problem 


•  ANSI  adopted  the  European  Union  Directive  in 
ANSI  S2.70  (2006),  but  it  does  not  contain 
specific  criteria  as  does  the  ANSI  S2.73  for  AVG 

•  There  are  no  US  regulations  for  manufacturers 
to  test,  certify,  and  label  power  tools 

•  Limited  prior  customer  input  to  GSA/DLA  for 
reduced  vibration  or  noise 
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Power  Tools:  The  Approach 


•  Evaluate  power  hand  tools  where  vibration  is  a  hazard 

•  Establish  procedures  for  the  Qualified  Products  List 

(QPL) 

•  Evaluate  possible  approaches  to  facilitate  and 
document  labs  which  can  provide  testing  and 
evaluation 

•  Crosslink  GSA,  DLA,  and  NIOSH  websites 

•  Make  improved  products  available  via  GSA  schedule 
both  to  Federal  and  Federal  contractor  buyers 
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Power  Tool  Selection  Criteria  and  Request 

For  Vendors  Information 

•  3rd  party  report  of  transmitted  vibration 

-  Measured  in  accordance  with  ANSI  2.70  and  NIOSH  guidelines  under  standard, 
specified  conditions 

•  Air  blow  off  directed  away  from  hands 

•  Other  ergonomic  criteria  (somewhat  dependent  on  product) 

-  Weight  -  balance  -  grip  dimensions  of  handle 

-  Surface  area  and  force  of  trigger 

-  Recoil  or  impulse  (different  than  “steady  state”  vibration) 

-  Wrist  deviation  associated  with  use 

•  Consistency  with  design  guidance,  noise  and  vibration  to 
be  weighted  factors  in  selection 

-  Minimum  eligibility  criteria  likely  to  be  established  for  the  Qualified  Products  List 
(QPL)  for  specific  equipment  and  products 

-  Data  may  be  reported  in  item  description  and  reflected  in  GSA,  DLA  and 
safety/health  websites 

•  Consider  warning  labels  as  needed  re:  noise  and 

vibration  20 


New  Tools  in  Federal  Supply  System 

GSA  is  continuing  to  incorporate  low  vibration  and  other  ergonomic 
characteristics  into  procurement  criteria  for  new  and  updated  power  hand  tools 
Pneumatic  riveting  hammer,  described  as  HAMMER,  PNEUMATIC,  PORTABLE 
5130-01-5716908. 

-  Its  vibration  (<2.5  m/s2)  is  less  than  half  the  level  created  by  many  legacy 
tools. 

Pneumatic  reciprocating  saw,  listed  as  SAW,  RECIPROCATING,  PNEUMATIC 
5130-01-572-5529. 

-  Its  vibration  (<4  m/s2)  is  less  than  half  the  level  created  by  many  legacy 
tools. 

Needle  scaler  (needle  gun),  listed  as  SCALER,  PNEUMATIC,  PORTABLE  5130- 
01-317-2453. 

-  To  date,  GSA  has  been  unable  to  specify  a  maximum  vibration  level  for  this 
tool. 

-  However,  one  vendor's  product,  which  served  as  a  guide  for  the  item 
specification,  reportedly  had  vibration  levels  in  the  range  of  3.5  m/s,  also 
considerably  lower  than  many  legacy  products. 

Continued  availability  will  depend  on  demand!  21 


Challenges 

Educating  industrial  hygienists  to  understand  and 
engage  in  existing  processes  for  feedback  and  glove  and 
tool  improvement 

Educating  safety  and  industrial  hygiene  managers  to 
understand  the  importance  of  improving  workers  gloves 
and  tools  as  opposed  to  traditional  surveys  and  reports 
Streamlining  and  clarifying  current  processes  and 
policies 

Incorporating  risk  management  in  glove  and  tool 
selection 

-  Involves  identifying  and  communicating  with 
responsible  technical  authorities  and  program  offices 

Communication  99 


Questions? 
Dina  Koza,  NA  VAIR 
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